--- title: "Demonstrate-Search-Predict — Khattab 2022" description: "Pairing a frozen language model with a retriever usually follows a fixed recipe: fetch passages for the query, insert them into the prompt, and let the…" authors: ["Omar Khattab", "Keshav Santhanam", "Xiang Lisa Li", "David Hall", "Percy Liang", "Christopher Potts", "Matei Zaharia"] year: 2022 venue: "arXiv:2212.14024" source: "https://arxiv.org/abs/2212.14024" doi: "10.48550/arxiv.2212.14024" arxiv: "2212.14024" licence: "arXiv — non-exclusive license to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/khattab-2022-demonstrate-search-predict-composing-retrieval/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/khattab-2022-demonstrate-search-predict-composing-retrieval/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP # Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP **Authors:** [Omar Khattab](https://md.research.elamur.ai/free/en/authors/omar-khattab/), [Keshav Santhanam](https://md.research.elamur.ai/free/en/authors/keshav-santhanam/), [Xiang Lisa Li](https://md.research.elamur.ai/free/en/authors/xiang-lisa-li/), [David Hall](https://md.research.elamur.ai/free/en/authors/david-hall/), [Percy Liang](https://md.research.elamur.ai/free/en/authors/percy-liang/), [Christopher Potts](https://md.research.elamur.ai/free/en/authors/christopher-potts/), [Matei Zaharia](https://md.research.elamur.ai/free/en/authors/matei-zaharia/) · **Year:** 2022 · **Venue:** arXiv:2212.14024 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2212.14024) · [PDF](https://arxiv.org/pdf/2212.14024) · [DOI 10.48550/arxiv.2212.14024](https://doi.org/10.48550/arxiv.2212.14024) · [arXiv 2212.14024](https://arxiv.org/abs/2212.14024) · [OpenAlex](https://openalex.org/W4313304293) **Citations:** 51 citations (openalex-snapshot · 2026-10-14) · [160](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Pairing a frozen language model with a retriever usually follows a fixed recipe: fetch passages for the query, insert them into the prompt, and let the model answer. This fails when no single passage holds the answer, as with multi-hop or conversational questions. The authors propose DSP, a Python framework in which an LM and an RM exchange natural-language text inside short, task-specific programs built from composable operations. DEMONSTRATE builds few-shot examples by running the pipeline on labeled question–answer pairs and keeping only the runs that reach the correct answer. This means intermediate steps, such as search queries, need no manual annotation. SEARCH lets the LM formulate queries across several retrieval hops, resolve conversational context and condense what earlier hops found. PREDICT generates the final answer from the gathered passages, optionally sampling several candidates and picking one by self-consistency. With GPT-3.5 and ColBERTv2 and no fine-tuning, DSP programs achieved new in-context learning bests on Open-SQuAD, HotPotQA and QReCC. This suggests that carefully designed pipelines over off-the-shelf models can be a cheap alternative to task-specific training. ## Key points - **Setup:** The LM is GPT-3.5 (text-davinci-002) and the RM is ColBERTv2, both used frozen without fine-tuning. The code is released at github.com/stanfordnlp/dsp. - **Overall gains:** Across tasks, DSP reports relative improvements of: - 37–120% over the vanilla LM, - 8–39% over a standard retrieve-then-read pipeline, - 80–290% over the contemporaneous self-ask pipeline. - **HotPotQA results:** In the open-domain "fullwiki" setting (Wikipedia 2017 abstracts), the multi-hop program reaches 51.4% EM. This beats the vanilla LM by 82%, retrieve-then-read by 39% and self-ask by 80% in EM. It also clearly surpasses concurrent methods such as a Wikipedia-API search system at 35.1% EM and PaLM-540B with self-consistency at 33.8% EM. - **QReCC evaluation:** QReCC is run in an open-domain setting over Wikipedia 2018. Conversations are evaluated auto-regressively, so the system sees its own earlier replies. The metric is novel-F1 (nF1), and noisy conversations are filtered out. - **Supervision:** Only end-task question–answer labels are required. Annotations for intermediate pipeline steps are bootstrapped automatically through weak supervision. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 160 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 140 not yet. - [Quiet-STaR: Language models can teach themselves to think before speaking](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/) - [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://md.research.elamur.ai/free/en/papers/chen-2023-frugalgpt-use-large-language-models/) - [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://md.research.elamur.ai/free/en/papers/es-2023-ragas-automated-evaluation-retrieval-augmented/) - [A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models](https://md.research.elamur.ai/free/en/papers/fan-2024-survey-rag-meeting-llms-towards/) - [Retrieval-Augmented Generation for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/) - [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/) - [Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity](https://md.research.elamur.ai/free/en/papers/jeong-2024-adaptive-rag-learning-adapt-retrieval/) - [Challenges and Applications of Large Language Models](https://md.research.elamur.ai/free/en/papers/kaddour-2023-challenges-applications-large-language-models/) - [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://md.research.elamur.ai/free/en/papers/khattab-2023-dspy-compiling-declarative-language-model/) - [Query Rewriting in Retrieval-Augmented Large Language Models](https://md.research.elamur.ai/free/en/papers/ma-2023-query-rewriting-retrieval-augmented-large/) - [The Prompt Report: A Systematic Survey of Prompt Engineering Techniques](https://md.research.elamur.ai/free/en/papers/schulhoff-2024-prompt-report-systematic-survey-prompt/) - [Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy](https://md.research.elamur.ai/free/en/papers/shao-2023-enhancing-retrieval-augmented-large-language/) - [Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models](https://md.research.elamur.ai/free/en/papers/shao-2024-assisting-writing-wikipedia-like-articles/) - [REPLUG: Retrieval-Augmented Black-Box Language Models](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/) - [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://md.research.elamur.ai/free/en/papers/tang-2023-toolalpaca-generalized-tool-learning-language/) - [Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions](https://md.research.elamur.ai/free/en/papers/trivedi-2022-interleaving-retrieval-chain-thought-reasoning/) - [FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation](https://md.research.elamur.ai/free/en/papers/vu-2023-freshllms-refreshing-large-language-models/) - [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://md.research.elamur.ai/free/en/papers/yoran-2023-making-retrieval-augmented-language-models/) - [Context-faithful Prompting for Large Language Models](https://md.research.elamur.ai/free/en/papers/zhou-2023-context-faithful-prompting-large-language/) - [Large Language Models for Information Retrieval: A Survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{khattab2022demonstrate, title = {{Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP}}, author = {Khattab, Omar and Santhanam, Keshav and Li, Xiang Lisa and Hall, David and Liang, Percy and Potts, Christopher and Zaharia, Matei}, year = {2022}, doi = {10.48550/arxiv.2212.14024}, eprint = {2212.14024}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2212.14024} } ``` - [← previous DEGREE: A Data-Efficient Generation-Based Event Extraction Model](https://md.research.elamur.ai/free/en/papers/hsu-2022-degree-data-efficient-generation-based/) - [next → Design Choices in Crowdsourcing Discourse Relation Annotations: The Effect of Worker Selection and Training](https://md.research.elamur.ai/free/en/papers/scholman-2022-design-choices-crowdsourcing-discourse-relation/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) ## References 62 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., and Chappidi, S. Open-domain question answering goes conversational via question rewriting. arXiv preprint arXiv:2010.04898, 2020. [find](https://scholar.google.com/scholar?q=Anantha%2C+R%2E%2C+Vakulenko%2C+S%2E%2C+Tu%2C+Z%2E%2C+Longpre%2C+S%2E%2C+Pulman%2C+S%2E%2C+and+Chappidi%2C+S%2E+Open%2Ddomain+question+answering+goes+conversational+via+question+rewriting%2E+arXiv+preprint+arXiv%3A2010%2E04898%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/anantha-2021-open-domain-question-answering-goes/) 2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. [find](https://scholar.google.com/scholar?q=Brown%2C+T%2E%2C+Mann%2C+B%2E%2C+Ryder%2C+N%2E%2C+Subbiah%2C+M%2E%2C+Kaplan%2C+J%2E+D%2E%2C+Dhariwal%2C+P%2E%2C+Neelakantan%2C+A%2E%2C+Shyam%2C+P%2E%2C+Sastry%2C+G%2E%2C+Askell%2C+A%2E%2C+et+al%2E+Language+models+are+few%2Dshot+learners%2E+Advances+in+neural+informati) 3. Cao, G., Nie, J.-Y., Gao, J., and Robertson, S. Selecting good expansion terms for pseudo-relevance feedback. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 243–250, 2008. [find](https://scholar.google.com/scholar?q=Cao%2C+G%2E%2C+Nie%2C+J%2E%2DY%2E%2C+Gao%2C+J%2E%2C+and+Robertson%2C+S%2E+Selecting+good+expansion+terms+for+pseudo%2Drelevance+feedback%2E+In+Proceedings+of+the+31st+annual+international+ACM+SIGIR+conference+on+Research+and+devel) 4. Chen, D., Fisch, A., Weston, J., and Bordes, A. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870–1879, Vancouver, Canada, 2017. Association for Computational Linguistics. do… [find](https://scholar.google.com/scholar?q=Chen%2C+D%2E%2C+Fisch%2C+A%2E%2C+Weston%2C+J%2E%2C+and+Bordes%2C+A%2E+Reading+Wikipedia+to+answer+open%2Ddomain+questions%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A) [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/) 5. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. [find](https://scholar.google.com/scholar?q=Chowdhery%2C+A%2E%2C+Narang%2C+S%2E%2C+Devlin%2C+J%2E%2C+Bosma%2C+M%2E%2C+Mishra%2C+G%2E%2C+Roberts%2C+A%2E%2C+Barham%2C+P%2E%2C+Chung%2C+H%2E+W%2E%2C+Sutton%2C+C%2E%2C+Gehrmann%2C+S%2E%2C+et+al%2E+Palm%3A+Scaling+language+modeling+with+pathways%2E+arXiv+preprint+arXi) 6. Del Tredici, M., Barlacchi, G., Shen, X., Cheng, W., and de Gispert, A. Question rewriting for open-domain conversational qa: Best practices and limitations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2974–2978, 2021. [find](https://scholar.google.com/scholar?q=Del+Tredici%2C+M%2E%2C+Barlacchi%2C+G%2E%2C+Shen%2C+X%2E%2C+Cheng%2C+W%2E%2C+and+de+Gispert%2C+A%2E+Question+rewriting+for+open%2Ddomain+conversational+qa%3A+Best+practices+and+limitations%2E+In+Proceedings+of+the+30th+ACM+Internation) 7. Dohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., et al. Language model cascades. arXiv preprint arXiv:2207.10342, 2022. [find](https://scholar.google.com/scholar?q=Dohan%2C+D%2E%2C+Xu%2C+W%2E%2C+Lewkowycz%2C+A%2E%2C+Austin%2C+J%2E%2C+Bieber%2C+D%2E%2C+Lopes%2C+R%2E+G%2E%2C+Wu%2C+Y%2E%2C+Michalewski%2C+H%2E%2C+Saurous%2C+R%2E+A%2E%2C+Sohl%2DDickstein%2C+J%2E%2C+et+al%2E+Language+model+cascades%2E+arXiv+preprint+arXiv%3A2207%2E10342%2C+20) 8. Fox, E. A. and Shaw, J. A. Combination of multiple searches. NIST special publication SP, 243, 1994. [find](https://scholar.google.com/scholar?q=Fox%2C+E%2E+A%2E+and+Shaw%2C+J%2E+A%2E+Combination+of+multiple+searches%2E+NIST+special+publication+SP%2C+243%2C+1994%2E) 9. Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V. Y., Lao, N., Lee, H., Juan, D.-C., et al. Attributed text generation via post-hoc research and revision. arXiv preprint arXiv:2210.08726, 2022. [find](https://scholar.google.com/scholar?q=Gao%2C+L%2E%2C+Dai%2C+Z%2E%2C+Pasupat%2C+P%2E%2C+Chen%2C+A%2E%2C+Chaganty%2C+A%2E+T%2E%2C+Fan%2C+Y%2E%2C+Zhao%2C+V%2E+Y%2E%2C+Lao%2C+N%2E%2C+Lee%2C+H%2E%2C+Juan%2C+D%2E%2DC%2E%2C+et+al%2E+Attributed+text+generation+via+post%2Dhoc+research+and+revision%2E+arXiv+preprint+arXi) [our card](https://md.research.elamur.ai/free/en/papers/gao-2023-rarr-researching-revising-what-language/) 10. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9: 346–361, 2021. [find](https://scholar.google.com/scholar?q=Geva%2C+M%2E%2C+Khashabi%2C+D%2E%2C+Segal%2C+E%2E%2C+Khot%2C+T%2E%2C+Roth%2C+D%2E%2C+and+Berant%2C+J%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+the+Association+for) 11. Hofstätter, S., Chen, J., Raman, K., and Zamani, H. Fid-light: Efficient and effective retrieval-augmented text generation. arXiv preprint arXiv:2209.14290, 2022. — [arXiv:2209.14290](https://arxiv.org/abs/2209.14290) [find](https://scholar.google.com/scholar?q=Hofst%C3%A4tter%2C+S%2E%2C+Chen%2C+J%2E%2C+Raman%2C+K%2E%2C+and+Zamani%2C+H%2E+Fid%2Dlight%3A+Efficient+and+effective+retrieval%2Daugmented+text+generation%2E+arXiv+preprint+arXiv%3A2209%2E14290%2C+2022%2E+%E2%80%94+%5BarXiv%3A2209%2E14290%5D%28https%3A%2F%2Farxiv%2Eor) 12. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022. [find](https://scholar.google.com/scholar?q=Huang%2C+J%2E%2C+Gu%2C+S%2E+S%2E%2C+Hou%2C+L%2E%2C+Wu%2C+Y%2E%2C+Wang%2C+X%2E%2C+Yu%2C+H%2E%2C+and+Han%2C+J%2E+Large+language+models+can+self%2Dimprove%2E+arXiv+preprint+arXiv%3A2210%2E11610%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/) 13. Ishii, Y., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. ACM Comput. Surv, 1(1), 2022. [find](https://scholar.google.com/scholar?q=Ishii%2C+Y%2E%2C+Madotto%2C+A%2E%2C+and+Fung%2C+P%2E+Survey+of+hallucination+in+natural+language+generation%2E+ACM+Comput%2E+Surv%2C+1%281%29%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/) 14. Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020. [find](https://scholar.google.com/scholar?q=Izacard%2C+G%2E+and+Grave%2C+E%2E+Leveraging+passage+retrieval+with+generative+models+for+open+domain+question+answering%2E+arXiv+preprint+arXiv%3A2007%2E01282%2C+2020%2E) [our card](https://md.research.elamur.ai/free/en/papers/izacard-2021-leveraging-passage-retrieval-generative-models/) 15. Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022. [find](https://scholar.google.com/scholar?q=Izacard%2C+G%2E%2C+Lewis%2C+P%2E%2C+Lomeli%2C+M%2E%2C+Hosseini%2C+L%2E%2C+Petroni%2C+F%2E%2C+Schick%2C+T%2E%2C+Dwivedi%2DYu%2C+J%2E%2C+Joulin%2C+A%2E%2C+Riedel%2C+S%2E%2C+and+Grave%2C+E%2E+Few%2Dshot+learning+with+retrieval+augmented+language+models%2E+arXiv+prepr) 16. Jiang, Y., Bordia, S., Zhong, Z., Dognin, C., Singh, M., and Bansal, M. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3441–3460, Online, 2020. Association for Computational Linguistics. doi: 10.1865… [find](https://scholar.google.com/scholar?q=Jiang%2C+Y%2E%2C+Bordia%2C+S%2E%2C+Zhong%2C+Z%2E%2C+Dognin%2C+C%2E%2C+Singh%2C+M%2E%2C+and+Bansal%2C+M%2E+HoVer%3A+A+dataset+for+many%2Dhop+fact+extraction+and+claim+verification%2E+In+Findings+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-hover-dataset-many-hop-fact/) 17. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781, Online, 2020. Association for Comput… [find](https://scholar.google.com/scholar?q=Karpukhin%2C+V%2E%2C+Oguz%2C+B%2E%2C+Min%2C+S%2E%2C+Lewis%2C+P%2E%2C+Wu%2C+L%2E%2C+Edunov%2C+S%2E%2C+Chen%2C+D%2E%2C+and+Yih%2C+W%2E%2Dt%2E+Dense+passage+retrieval+for+open%2Ddomain+question+answering%2E+In+Proceedings+of+the+2020+Conference+on+Empirical) [our card](https://md.research.elamur.ai/free/en/papers/karpukhin-2020-dense-passage-retrieval-open-domain/) 18. Khattab, O. and Zaharia, M. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Huang, J., Chang, Y., Cheng, X., Kamps, J., Murdock, V., Wen, J., and Liu, Y. (eds.), Proceedings of the 43rd International ACM SIGIR conference on research and development i… [find](https://scholar.google.com/scholar?q=Khattab%2C+O%2E+and+Zaharia%2C+M%2E+Colbert%3A+Efficient+and+effective+passage+search+via+contextualized+late+interaction+over+BERT%2E+In+Huang%2C+J%2E%2C+Chang%2C+Y%2E%2C+Cheng%2C+X%2E%2C+Kamps%2C+J%2E%2C+Murdock%2C+V%2E%2C+Wen%2C+J%2E%2C+and+Liu%2C) [our card](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/) 19. ACM, 2020. doi: 10.1145/3397271.3401075. URL https://doi.org/10.1145/3397271.3401075. [↗](https://doi.org/10.1145/3397271.3401075) [our card](https://md.research.elamur.ai/free/en/papers/khattab-2020-colbert-efficient-effective-passage/) 20. Khattab, O., Potts, C., and Zaharia, M. Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021a. [find](https://scholar.google.com/scholar?q=Khattab%2C+O%2E%2C+Potts%2C+C%2E%2C+and+Zaharia%2C+M%2E+Baleen%3A+Robust+Multi%2DHop+Reasoning+at+Scale+via+Condensed+Retrieval%2E+In+Thirty%2DFifth+Conference+on+Neural+Information+Processing+Systems%2C+2021a%2E) 21. Khattab, O., Potts, C., and Zaharia, M. Relevance-guided supervision for openqa with ColBERT. Transactions of the Association for Computational Linguistics, 9:929– 944, 2021b. [find](https://scholar.google.com/scholar?q=Khattab%2C+O%2E%2C+Potts%2C+C%2E%2C+and+Zaharia%2C+M%2E+Relevance%2Dguided+supervision+for+openqa+with+ColBERT%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+9%3A929%E2%80%93+944%2C+2021b%2E) 22. Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022. [find](https://scholar.google.com/scholar?q=Khot%2C+T%2E%2C+Trivedi%2C+H%2E%2C+Finlayson%2C+M%2E%2C+Fu%2C+Y%2E%2C+Richardson%2C+K%2E%2C+Clark%2C+P%2E%2C+and+Sabharwal%2C+A%2E+Decomposed+prompting%3A+A+modular+approach+for+solving+complex+tasks%2E+arXiv+preprint+arXiv%3A2210%2E02406%2C+2022%2E) 23. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. [find](https://scholar.google.com/scholar?q=Kojima%2C+T%2E%2C+Gu%2C+S%2E+S%2E%2C+Reid%2C+M%2E%2C+Matsuo%2C+Y%2E%2C+and+Iwasawa%2C+Y%2E+Large+language+models+are+zero%2Dshot+reasoners%2E+arXiv+preprint+arXiv%3A2205%2E11916%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/) 24. Krishna, K., Chang, Y., Wieting, J., and Iyyer, M. Rankgen: Improving text generation with large ranking models. arXiv preprint arXiv:2205.09726, 2022. [find](https://scholar.google.com/scholar?q=Krishna%2C+K%2E%2C+Chang%2C+Y%2E%2C+Wieting%2C+J%2E%2C+and+Iyyer%2C+M%2E+Rankgen%3A+Improving+text+generation+with+large+ranking+models%2E+arXiv+preprint+arXiv%3A2205%2E09726%2C+2022%2E) 25. Kurland, O. and Culpepper, J. S. Fusion in information retrieval: Sigir 2018 half-day tutorial. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1383–1386, 2018. [find](https://scholar.google.com/scholar?q=Kurland%2C+O%2E+and+Culpepper%2C+J%2E+S%2E+Fusion+in+information+retrieval%3A+Sigir+2018+half%2Dday+tutorial%2E+In+The+41st+International+ACM+SIGIR+Conference+on+Research+%26+Development+in+Information+Retrieval%2C+pp%2E+1) 26. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for question answering research. T… [find](https://scholar.google.com/scholar?q=Kwiatkowski%2C+T%2E%2C+Palomaki%2C+J%2E%2C+Redfield%2C+O%2E%2C+Collins%2C+M%2E%2C+Parikh%2C+A%2E%2C+Alberti%2C+C%2E%2C+Epstein%2C+D%2E%2C+Polosukhin%2C+I%2E%2C+Devlin%2C+J%2E%2C+Lee%2C+K%2E%2C+Toutanova%2C+K%2E%2C+Jones%2C+L%2E%2C+Kelcey%2C+M%2E%2C+Chang%2C+M%2E%2DW%2E%2C+Dai%2C+A%2E+M%2E%2C+Usz) [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2019-natural-questions-benchmark-question-answering/) 27. Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115, 2022. [find](https://scholar.google.com/scholar?q=Lazaridou%2C+A%2E%2C+Gribovskaya%2C+E%2E%2C+Stokowiec%2C+W%2E%2C+and+Grigorev%2C+N%2E+Internet%2Daugmented+language+models+through+few%2Dshot+prompting+for+open%2Ddomain+question+answering%2E+arXiv+preprint+arXiv%3A2203%2E05115%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/lazaridou-2022-internet-augmented-language-models-through/) 28. Le, N. T., Bai, F., and Ritter, A. Few-shot anaphora resolution in scientific protocols via mixtures of in-context experts. arXiv preprint arXiv:2210.03690, 2022. [find](https://scholar.google.com/scholar?q=Le%2C+N%2E+T%2E%2C+Bai%2C+F%2E%2C+and+Ritter%2C+A%2E+Few%2Dshot+anaphora+resolution+in+scientific+protocols+via+mixtures+of+in%2Dcontext+experts%2E+arXiv+preprint+arXiv%3A2210%2E03690%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/le-2022-few-shot-anaphora-resolution-scientific/) 29. Lee, K., Chang, M.-W., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086– 6096, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/… [find](https://scholar.google.com/scholar?q=Lee%2C+K%2E%2C+Chang%2C+M%2E%2DW%2E%2C+and+Toutanova%2C+K%2E+Latent+retrieval+for+weakly+supervised+open+domain+question+answering%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguisti) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-latent-retrieval-weakly-supervised-open/) 30. Levine, Y., Dalmedigos, I., Ram, O., Zeldes, Y., Jannai, D., Muhlgay, D., Osin, Y., Lieber, O., Lenz, B., Shalev-Shwartz, S., et al. Standing on the shoulders of giant frozen language models. arXiv preprint arXiv:2204.10019, 2022. — [arXiv:2204.10019](https://arxiv.org/abs/2204.10019) [find](https://scholar.google.com/scholar?q=Levine%2C+Y%2E%2C+Dalmedigos%2C+I%2E%2C+Ram%2C+O%2E%2C+Zeldes%2C+Y%2E%2C+Jannai%2C+D%2E%2C+Muhlgay%2C+D%2E%2C+Osin%2C+Y%2E%2C+Lieber%2C+O%2E%2C+Lenz%2C+B%2E%2C+Shalev%2DShwartz%2C+S%2E%2C+et+al%2E+Standing+on+the+shoulders+of+giant+frozen+language+models%2E+arXiv+pr) 31. Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Ad… [find](https://scholar.google.com/scholar?q=Lewis%2C+P%2E+S%2E+H%2E%2C+Perez%2C+E%2E%2C+Piktus%2C+A%2E%2C+Petroni%2C+F%2E%2C+Karpukhin%2C+V%2E%2C+Goyal%2C+N%2E%2C+K%C3%BCttler%2C+H%2E%2C+Lewis%2C+M%2E%2C+Yih%2C+W%2E%2C+Rockt%C3%A4schel%2C+T%2E%2C+Riedel%2C+S%2E%2C+and+Kiela%2C+D%2E+Retrieval%2DAugmented+Generation+for+Knowledge%2D) 32. Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022. [find](https://scholar.google.com/scholar?q=Li%2C+X%2E+L%2E%2C+Holtzman%2C+A%2E%2C+Fried%2C+D%2E%2C+Liang%2C+P%2E%2C+Eisner%2C+J%2E%2C+Hashimoto%2C+T%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Lewis%2C+M%2E+Contrastive+decoding%3A+Open%2Dended+text+generation+as+optimization%2E+arXiv+preprint+arXiv%3A2210%2E150) [our card](https://md.research.elamur.ai/free/en/papers/li-2022-contrastive-decoding-open-ended-text/) 33. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021. [find](https://scholar.google.com/scholar?q=Liu%2C+J%2E%2C+Shen%2C+D%2E%2C+Zhang%2C+Y%2E%2C+Dolan%2C+B%2E%2C+Carin%2C+L%2E%2C+and+Chen%2C+W%2E+What+makes+good+in%2Dcontext+examples+for+gpt%2D3%3F+arXiv+preprint+arXiv%3A2101%2E06804%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) 34. McCann, B., Keskar, N. S., Xiong, C., and Socher, R. The natural language decathlon: Multitask learning as question answering. arXiv:1806.08730, 2018. URL https://arxiv.org/abs/1806.08730. [↗](https://arxiv.org/abs/1806.08730) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2018-natural-language-decathlon-multitask-learning/) 35. Min, S., Zhong, V., Zettlemoyer, L., and Hajishirzi, H. Multi-hop reading comprehension through question decomposition and rescoring. arXiv preprint arXiv:1906.02916, 2019. [find](https://scholar.google.com/scholar?q=Min%2C+S%2E%2C+Zhong%2C+V%2E%2C+Zettlemoyer%2C+L%2E%2C+and+Hajishirzi%2C+H%2E+Multi%2Dhop+reading+comprehension+through+question+decomposition+and+rescoring%2E+arXiv+preprint+arXiv%3A1906%2E02916%2C+2019%2E) [our card](https://md.research.elamur.ai/free/en/papers/min-2019-multi-hop-reading-comprehension-through/) 36. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. [find](https://scholar.google.com/scholar?q=Ouyang%2C+L%2E%2C+Wu%2C+J%2E%2C+Jiang%2C+X%2E%2C+Almeida%2C+D%2E%2C+Wainwright%2C+C%2E+L%2E%2C+Mishkin%2C+P%2E%2C+Zhang%2C+C%2E%2C+Agarwal%2C+S%2E%2C+Slama%2C+K%2E%2C+Ray%2C+A%2E%2C+et+al%2E+Training+language+models+to+follow+instructions+with+human+feedback%2E+arXi) 37. Paranjape, A., Khattab, O., Potts, C., Zaharia, M., and Manning, C. D. Hindsight: Posterior-guided Training of Retrievers for Improved Open-ended Generation. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=Vr_BTpw3wz](https://openreview.net/forum?id=Vr_BTpw3wz). [↗](https://openreview.net/forum?id=Vr_BTpw3wz) 38. Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:11054–11070, 2021. [find](https://scholar.google.com/scholar?q=Perez%2C+E%2E%2C+Kiela%2C+D%2E%2C+and+Cho%2C+K%2E+True+few%2Dshot+learning+with+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%3A11054%E2%80%9311070%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/perez-2021-true-few-shot-learning-language/) 39. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. [find](https://scholar.google.com/scholar?q=Press%2C+O%2E%2C+Zhang%2C+M%2E%2C+Min%2C+S%2E%2C+Schmidt%2C+L%2E%2C+Smith%2C+N%2E+A%2E%2C+and+Lewis%2C+M%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03350%2C+2022%2E) 40. Qi, P., Lee, H., Sido, O., Manning, C. D., et al. Retrieve, rerank, read, then iterate: Answering open-domain questions of arbitrary complexity from text. arXiv preprint arXiv:2010.12527, 2020. URL https://arxiv.org/abs/2010.12527. — [arXiv:2010.12527](https://arxiv.org/abs/2010.12527) [↗](https://arxiv.org/abs/2010.12527) 41. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. [find](https://scholar.google.com/scholar?q=Radford%2C+A%2E%2C+Wu%2C+J%2E%2C+Child%2C+R%2E%2C+Luan%2C+D%2E%2C+Amodei%2C+D%2E%2C+Sutskever%2C+I%2E%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2C+2019%2E) 42. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383– 2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v… [find](https://scholar.google.com/scholar?q=Rajpurkar%2C+P%2E%2C+Zhang%2C+J%2E%2C+Lopyrev%2C+K%2E%2C+and+Liang%2C+P%2E+SQuAD%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natural+Language+Process) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) 43. Raposo, G., Ribeiro, R., Martins, B., and Coheur, L. Question rewriting? assessing its importance for conversational question answering. In European Conference on Information Retrieval, pp. 199–206. Springer, 2022. [find](https://scholar.google.com/scholar?q=Raposo%2C+G%2E%2C+Ribeiro%2C+R%2E%2C+Martins%2C+B%2E%2C+and+Coheur%2C+L%2E+Question+rewriting%3F+assessing+its+importance+for+conversational+question+answering%2E+In+European+Conference+on+Information+Retrieval%2C+pp%2E+199%E2%80%93206%2E+S) 44. Santhanam, K., Khattab, O., Potts, C., and Zaharia, M. PLAID: An Efficient Engine for Late Interaction Retrieval. arXiv preprint arXiv:2205.09707, 2022a. [find](https://scholar.google.com/scholar?q=Santhanam%2C+K%2E%2C+Khattab%2C+O%2E%2C+Potts%2C+C%2E%2C+and+Zaharia%2C+M%2E+PLAID%3A+An+Efficient+Engine+for+Late+Interaction+Retrieval%2E+arXiv+preprint+arXiv%3A2205%2E09707%2C+2022a%2E) 45. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., and Zaharia, M. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp… [find](https://scholar.google.com/scholar?q=Santhanam%2C+K%2E%2C+Khattab%2C+O%2E%2C+Saad%2DFalcon%2C+J%2E%2C+Potts%2C+C%2E%2C+and+Zaharia%2C+M%2E+ColBERTv2%3A+Effective+and+efficient+retrieval+via+lightweight+late+interaction%2E+In+Proceedings+of+the+2022+Conference+of+the+Nort) [our card](https://md.research.elamur.ai/free/en/papers/santhanam-2022-colbertv2-effective-efficient-retrieval-lightweight/) 46. Shuster, K., Poff, S., Chen, M., Kiela, D., and Weston, J. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021. [find](https://scholar.google.com/scholar?q=Shuster%2C+K%2E%2C+Poff%2C+S%2E%2C+Chen%2C+M%2E%2C+Kiela%2C+D%2E%2C+and+Weston%2C+J%2E+Retrieval+augmentation+reduces+hallucination+in+conversation%2E+arXiv+preprint+arXiv%3A2104%2E07567%2C+2021%2E) [our card](https://md.research.elamur.ai/free/en/papers/shuster-2021-retrieval-augmentation-reduces-hallucination-conversation/) 47. Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., and Wang, L. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022. [find](https://scholar.google.com/scholar?q=Si%2C+C%2E%2C+Gan%2C+Z%2E%2C+Yang%2C+Z%2E%2C+Wang%2C+S%2E%2C+Wang%2C+J%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+Wang%2C+L%2E+Prompting+gpt%2D3+to+be+reliable%2E+arXiv+preprint+arXiv%3A2210%2E09150%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/si-2022-prompting-gpt-3-reliable/) 48. Sun, Z., Wang, X., Tay, Y., Yang, Y., and Zhou, D. Recitation-augmented language models. arXiv preprint arXiv:2210.01296, 2022. — [arXiv:2210.01296](https://arxiv.org/abs/2210.01296) [find](https://scholar.google.com/scholar?q=Sun%2C+Z%2E%2C+Wang%2C+X%2E%2C+Tay%2C+Y%2E%2C+Yang%2C+Y%2E%2C+and+Zhou%2C+D%2E+Recitation%2Daugmented+language+models%2E+arXiv+preprint+arXiv%3A2210%2E01296%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E01296%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2210%2E01296%29) 49. Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)… [find](https://scholar.google.com/scholar?q=Thorne%2C+J%2E%2C+Vlachos%2C+A%2E%2C+Christodoulopoulos%2C+C%2E%2C+and+Mittal%2C+A%2E+FEVER%3A+a+large%2Dscale+dataset+for+fact+extraction+and+VERification%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+o) [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/) 50. Vakulenko, S., Kiesel, J., and Fröbe, M. SCAI-QReCC shared task on conversational question answering. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 4913–4922, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022… [↗](https://aclanthology.org/2022…) 51. Wang, X., Macdonald, C., Tonellotto, N., and Ounis, I. Colbert-prf: Semantic pseudo-relevance feedback for dense passage and document retrieval. ACM Transactions on the Web, 2022a. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Macdonald%2C+C%2E%2C+Tonellotto%2C+N%2E%2C+and+Ounis%2C+I%2E+Colbert%2Dprf%3A+Semantic+pseudo%2Drelevance+feedback+for+dense+passage+and+document+retrieval%2E+ACM+Transactions+on+the+Web%2C+2022a%2E) 52. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Rationale-augmented ensembles in language models. arXiv preprint arXiv:2207.00747, 2022b. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Wei%2C+J%2E%2C+Schuurmans%2C+D%2E%2C+Le%2C+Q%2E%2C+Chi%2C+E%2E%2C+and+Zhou%2C+D%2E+Rationale%2Daugmented+ensembles+in+language+models%2E+arXiv+preprint+arXiv%3A2207%2E00747%2C+2022b%2E) 53. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022c. [find](https://scholar.google.com/scholar?q=Wang%2C+X%2E%2C+Wei%2C+J%2E%2C+Schuurmans%2C+D%2E%2C+Le%2C+Q%2E%2C+Chi%2C+E%2E%2C+and+Zhou%2C+D%2E+Self%2Dconsistency+improves+chain+of+thought+reasoning+in+language+models%2E+arXiv+preprint+arXiv%3A2203%2E11171%2C+2022c%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/) 54. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%2C+Wang%2C+X%2E%2C+Schuurmans%2C+D%2E%2C+Bosma%2C+M%2E%2C+Chi%2C+E%2E%2C+Le%2C+Q%2E%2C+and+Zhou%2C+D%2E+Chain+of+thought+prompting+elicits+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2201%2E11903%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) 55. Wiher, G., Meister, C., and Cotterell, R. On decoding strategies for neural text generators. arXiv preprint arXiv:2203.15721, 2022. [find](https://scholar.google.com/scholar?q=Wiher%2C+G%2E%2C+Meister%2C+C%2E%2C+and+Cotterell%2C+R%2E+On+decoding+strategies+for+neural+text+generators%2E+arXiv+preprint+arXiv%3A2203%2E15721%2C+2022%2E) 56. Xiong, W., Li, X. L., Iyer, S., Du, J., Lewis, P., Wang, W. Y., Mehdad, Y., Yih, W.-t., Riedel, S., Kiela, D., et al. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint arXiv:2009.12756, 2020. URL https://arxiv.org/abs/2009.12756. — [arXiv:2009.12756](https://arxiv.org/abs/2009.12756) [↗](https://arxiv.org/abs/2009.12756) 57. Xue, X. and Croft, W. B. Modeling reformulation using query distributions. ACM Transactions on Information Systems (TOIS), 31(2):1–34, 2013. [find](https://scholar.google.com/scholar?q=Xue%2C+X%2E+and+Croft%2C+W%2E+B%2E+Modeling+reformulation+using+query+distributions%2E+ACM+Transactions+on+Information+Systems+%28TOIS%29%2C+31%282%29%3A1%E2%80%9334%2C+2013%2E) 58. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. [find](https://scholar.google.com/scholar?q=Yang%2C+Z%2E%2C+Qi%2C+P%2E%2C+Zhang%2C+S%2E%2C+Bengio%2C+Y%2E%2C+Cohen%2C+W%2E+W%2E%2C+Salakhutdinov%2C+R%2E%2C+and+Manning%2C+C%2E+D%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answering%2E+arXiv+preprint+arXiv%3A1809%2E09600%2C) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/) 59. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. [find](https://scholar.google.com/scholar?q=Yao%2C+S%2E%2C+Zhao%2C+J%2E%2C+Yu%2C+D%2E%2C+Du%2C+N%2E%2C+Shafran%2C+I%2E%2C+Narasimhan%2C+K%2E%2C+and+Cao%2C+Y%2E+React%3A+Synergizing+reasoning+and+acting+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03629%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/) 60. Zelikman, E., Wu, Y., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465, 2022. [find](https://scholar.google.com/scholar?q=Zelikman%2C+E%2E%2C+Wu%2C+Y%2E%2C+and+Goodman%2C+N%2E+D%2E+Star%3A+Bootstrapping+reasoning+with+reasoning%2E+arXiv+preprint+arXiv%3A2203%2E14465%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/) 61. Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022. [find](https://scholar.google.com/scholar?q=Zhang%2C+Z%2E%2C+Zhang%2C+A%2E%2C+Li%2C+M%2E%2C+and+Smola%2C+A%2E+Automatic+chain+of+thought+prompting+in+large+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03493%2C+2022%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2022-automatic-chain-thought-prompting-large/) 62. Zhong, V., Shi, W., Yih, W.-t., and Zettlemoyer, L. Romqa: A benchmark for robust, multi-evidence, multi-answer question answering. arXiv preprint arXiv:2210.14353, 2022. [find](https://scholar.google.com/scholar?q=Zhong%2C+V%2E%2C+Shi%2C+W%2E%2C+Yih%2C+W%2E%2Dt%2E%2C+and+Zettlemoyer%2C+L%2E+Romqa%3A+A+benchmark+for+robust%2C+multi%2Devidence%2C+multi%2Danswer+question+answering%2E+arXiv+preprint+arXiv%3A2210%2E14353%2C+2022%2E)