---
title: "Iteratively Prompt Pre-trained Language Models for Chain of"
description: "The authors propose iterative prompting. 47 references, cited by 71 papers here. Among them: 73."
authors: ["Boshi Wang", "Xiang Deng", "Huan Sun"]
year: 2022
venue: "arXiv:2203.08383"
source: "https://arxiv.org/abs/2203.08383"
doi: "10.48550/arxiv.2203.08383"
arxiv: "2203.08383"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2022-iteratively-prompt-pre-trained-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2022-iteratively-prompt-pre-trained-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/) / Iteratively Prompt Pre-trained Language Models for Chain of Thought

# Iteratively Prompt Pre-trained Language Models for Chain of Thought

**Authors:** [Boshi Wang](https://md.research.elamur.ai/free/en/authors/boshi-wang/), [Xiang Deng](https://md.research.elamur.ai/free/en/authors/xiang-deng/) and [Huan Sun](https://md.research.elamur.ai/free/en/authors/huan-sun/) · **Year:** 2022 · **Venue:** [arXiv:2203.08383](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2203.08383) · [PDF](https://arxiv.org/pdf/2203.08383) · [DOI 10.48550/arxiv.2203.08383](https://doi.org/10.48550/arxiv.2203.08383) · [arXiv 2203.08383](https://arxiv.org/abs/2203.08383) · [OpenAlex](https://openalex.org/W4385572867)

**Citations:** 73 citations (OpenAlex, 2026-10-14) · [71](https://md.research.elamur.ai/free/en/papers/wang-2022-iteratively-prompt-pre-trained-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Pre-trained language models (PLMs) hold many simple facts but often fail on questions that require chaining several of them, such as finding a person's father and then that father's birthplace. The authors propose iterative prompting. The PLM stays frozen and recalls one knowledge statement per step, and each step's input combines the query with the statements recalled so far. Earlier prompting methods either assume one identifiable relation per query or learn static task-level prompts that ignore what changes between steps. To address this, the paper introduces a Context-Aware Prompter. This transformer model contextualizes special tokens against the current step's input and turns them into continuous prompts, which are prepended to the encoder and decoder of a fixed BART-large. The prompter was evaluated on 2WikiMultiHopQA, R4C and a taxonomic reasoning dataset (LoT). It outperforms Prompt-Tuning, Prefix-Tuning and other baselines by notable margins, and further analysis indicates its prompting behaviour is faithful. The approach matters because it enables step-wise reasoning in moderately sized models without changing their weights. Chain-of-thought prompting, by contrast, usually works well only in models with more than 100B parameters.

## Key points
- **Datasets:** three were used: 2WikiMultiHopQA from Ho et al. (2020), R4C from Inoue et al. (2020), and LoT, a taxonomic-relation dataset from Talmor et al. (2020b). Attempts with HotpotQA and StrategyQA failed because their knowledge statements were too noisy.
- **Training and inference:** the backbone is a frozen BART-large (encoder-decoder) with continuous prefix prompts. Training uses teacher forcing with seq2seq objectives. Inference is autoregressive over a fixed number of steps, with an optional learned "stopper" that ends recall early.
- **Trainable parameters:** the prompter needs Θ(d·(D+l)), compared with Θ(l·D) for Prompt-Tuning and Θ(l·h·D) for Prefix-Tuning. Its cost therefore grows no faster than these alternatives as the PLM scales.
- **Motivation:** fine-tuning large PLMs is costly and may damage the knowledge they store (catastrophic forgetting). Generating all facts in one pass tends to produce irrelevant or hallucinated statements.
- **Code:** released at github.com/sunlab-osu/IterPrompt. Proposed future directions include using the idea in pretraining and letting humans correct intermediate reasoning steps.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 90 works · 19,112 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Boshi Wang, Xiang Deng and Huan Sun. 2022. Iteratively Prompt Pre-trained Language Models for Chain of Thought. arXiv:2203.08383.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2022iteratively,
  title = {{Iteratively Prompt Pre-trained Language Models for Chain of Thought}},
  author = {Boshi Wang, Xiang Deng and Huan Sun},
  year = {2022},
  doi = {10.48550/arxiv.2203.08383},
  eprint = {2203.08383},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2203.08383}
}
```

- [← previous InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://md.research.elamur.ai/free/en/papers/wang-2022-internvideo-general-video-foundation/)
- [next → A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation](https://md.research.elamur.ai/free/en/papers/liu-2022-token-level-reference-free-hallucination/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/3/)

## References

47 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021. [Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training](https://md.research.elamur.ai/free/en/papers/agarwal-2021-knowledge-graph-based-synthetic-corpus/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
3. Ohio Supercomputer Center. 1987. Ohio supercomputer center. [find](https://scholar.google.com/scholar?q=Ohio+Supercomputer+Center%2E+1987%2E+Ohio+supercomputer+center%2E)
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
5. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346–361. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Daniel+Khashabi%2C+Elad+Segal%2C+Tushar+Khot%2C+Dan+Roth%2C+and+Jonathan+Berant%2E+2021%2E+Did+aristotle+use+a+laptop%3F+a+question+answering+benchmark+with+implicit+reasoning+strategies%2E+Transactions+of+)
6. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. [Retrieval augmented language model pre-training](https://md.research.elamur.ai/free/en/papers/guu-2020-realm-retrieval-augmented-language-model/). In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 3929–3938. PMLR.
7. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association f… [find](https://scholar.google.com/scholar?q=John+Hewitt+and+Percy+Liang%2E+2019%2E+Designing+and+interpreting+probes+with+control+tasks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the+9th+Internati)
8. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. [Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps](https://md.research.elamur.ai/free/en/papers/ho-2020-constructing-multi-hop-qa-dataset/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
9. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
10. Naoya Inoue, Pontus Stenetorp, and Kentaro Inui. 2020. R4C: A benchmark for evaluating RC systems to get the right answer for the right reason. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6740–6750, Online. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Naoya+Inoue%2C+Pontus+Stenetorp%2C+and+Kentaro+Inui%2E+2020%2E+R4C%3A+A+benchmark+for+evaluating+RC+systems+to+get+the+right+answer+for+the+right+reason%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associa)
11. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438.
12. Nora Kassner, Benno Krojer, and Hinrich Schütze. 2020. [Are pretrained language models symbolic reasoners over knowledge?](https://md.research.elamur.ai/free/en/papers/kassner-2020-are-pretrained-language-models-symbolic/) In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 552–564, Online. Association for Computational Linguistics.
13. Omar Khattab, Christopher Potts, and Matei Zaharia. 2021. Baleen: Robust multi-hop reasoning at scale via condensed retrieval. Advances in Neural Information Processing Systems, 34. [find](https://scholar.google.com/scholar?q=Omar+Khattab%2C+Christopher+Potts%2C+and+Matei+Zaharia%2E+2021%2E+Baleen%3A+Robust+multi%2Dhop+reasoning+at+scale+via+condensed+retrieval%2E+Advances+in+Neural+Information+Processing+Systems%2C+34%2E)
14. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks… [find](https://scholar.google.com/scholar?q=James+Kirkpatrick%2C+Razvan+Pascanu%2C+Neil+Rabinowitz%2C+Joel+Veness%2C+Guillaume+Desjardins%2C+Andrei+A%2E+Rusu%2C+Kieran+Milan%2C+John+Quan%2C+Tiago+Ramalho%2C+Agnieszka+Grabska%2DBarwinska%2C+Demis+Hassabis%2C+Claudia+Clop)
15. Nayeon Lee, Belinda Z. Li, Sinong Wang, Wen-tau Yih, Hao Ma, and Madian Khabsa. 2020. Language models as fact checkers? In Proceedings of the Third Workshop on Fact Extraction and VERification (FEVER), pages 36–41, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nayeon+Lee%2C+Belinda+Z%2E+Li%2C+Sinong+Wang%2C+Wen%2Dtau+Yih%2C+Hao+Ma%2C+and+Madian+Khabsa%2E+2020%2E+Language+models+as+fact+checkers%3F+In+Proceedings+of+the+Third+Workshop+on+Fact+Extraction+and+VERification+%28FEVER%29)
16. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist…
17. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
18. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1000–1008, Online. Associ… [find](https://scholar.google.com/scholar?q=Patrick+Lewis%2C+Pontus+Stenetorp%2C+and+Sebastian+Riedel%2E+2021%2E+Question+and+answer+test%2Dtrain+overlap+in+open%2Ddomain+question+answering+datasets%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Ch)
19. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
20. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). arXiv preprint arXiv:2107.13586.
21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
22. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+International+Conference+on+Learning+Representations%2E)
23. Michael McCloskey and Neal J. Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower, editor, Psychology of Learning and Motivation, volume 24, pages 109–165. Academic Press. [find](https://scholar.google.com/scholar?q=Michael+McCloskey+and+Neal+J%2E+Cohen%2E+1989%2E+Catastrophic+interference+in+connectionist+networks%3A+The+sequential+learning+problem%2E+In+Gordon+H%2E+Bower%2C+editor%2C+Psychology+of+Learning+and+Motivation%2C+volu)
24. Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021. [Reframing instructional prompts to gptk’s language](https://md.research.elamur.ai/free/en/papers/mishra-2022-reframing-instructional-prompts-gptk-s/). arXiv preprint arXiv:2109.07830.
25. Lingbo Mo, Ashley Lewis, Huan Sun, and Michael White. 2022a. Towards transparent interactive semantic parsing via step-by-step correction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 322–342. [find](https://scholar.google.com/scholar?q=Lingbo+Mo%2C+Ashley+Lewis%2C+Huan+Sun%2C+and+Michael+White%2E+2022a%2E+Towards+transparent+interactive+semantic+parsing+via+step%2Dby%2Dstep+correction%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A)
26. Lingbo Mo, Zhen Wang, Jie Zhao, and Huan Sun. 2022b. Knowledge transfer between structured and unstructured sources for complex question answering. In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pages 55–66. [find](https://scholar.google.com/scholar?q=Lingbo+Mo%2C+Zhen+Wang%2C+Jie+Zhao%2C+and+Huan+Sun%2E+2022b%2E+Knowledge+transfer+between+structured+and+unstructured+sources+for+complex+question+answering%2E+In+Proceedings+of+the+Workshop+on+Structured+and+Uns)
27. Bhargavi Paranjape, Julian Michael, Marjan Ghazvininejad, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. Prompting contrastive explanations for commonsense reasoning tasks. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4179–4192, Online. Association for Compu… [find](https://scholar.google.com/scholar?q=Bhargavi+Paranjape%2C+Julian+Michael%2C+Marjan+Ghazvininejad%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2021%2E+Prompting+contrastive+explanations+for+commonsense+reasoning+tasks%2E+In+Findings+of+the+Associ)
28. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
29. Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D. Manning. 2019. [Answering complex open-domain questions through iterative query generation](https://md.research.elamur.ai/free/en/papers/qi-2019-answering-complex-open-domain-questions/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural L…
30. Guanghui Qin and Jason Eisner. 2021. [Learning how to ask: Querying LMs with mixtures of soft prompts](https://md.research.elamur.ai/free/en/papers/qin-2021-learning-ask-querying-lms-mixtures/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203–5212, Online. Association for Computatio…
31. Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446. [find](https://scholar.google.com/scholar?q=Jack+W+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+et+al%2E+2021%2E+Scaling+language+models%3A+Methods%2C+a)
32. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5418–5426, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Roberts%2C+Colin+Raffel%2C+and+Noam+Shazeer%2E+2020%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Langu)
33. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in bertology: What we know about how bert works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8:842–866.
34. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
35. Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4615–4629, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2C+Peter+West%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Unsupervised+commonsense+question+answering+with+self%2Dtalk%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Meth)
36. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020a. [olmpics-on what language model pre-training captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/). Transactions of the Association for Computational Linguistics, 8:743–758.
37. Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, and Jonathan Berant. 2020b. [Leap-of-thought: Teaching pre-trained models to systematically reason over implicit knowledge](https://md.research.elamur.ai/free/en/papers/talmor-2020-leap-thought-teaching-pre-trained/). In Advances in Neural Information Processing Systems, volume 33, pages 20227–20237.
38. Jesse Vig. 2019. [A multiscale visualization of attention in the transformer model](https://md.research.elamur.ai/free/en/papers/vig-2019-multiscale-visualization-attention-transformer-model/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37–42, Florence, Italy. Association for Computational Linguistics.
39. Cunxiang Wang, Pai Liu, and Yue Zhang. 2021. Can generative pre-trained language models serve as knowledge bases for closed-book QA? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (… [find](https://scholar.google.com/scholar?q=Cunxiang+Wang%2C+Pai+Liu%2C+and+Yue+Zhang%2E+2021%2E+Can+generative+pre%2Dtrained+language+models+serve+as+knowledge+bases+for+closed%2Dbook+QA%3F+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Co)
40. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on M…
41. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Xuezhi+Wang%2C+Dale+Schuurmans%2C+Maarten+Bosma%2C+Ed+Chi%2C+Quoc+Le%2C+and+Denny+Zhou%2E+2022b%2E)
42. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903. — [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/)
43. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — [Transformers: State-of-the-Art Natural Language Processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
44. Wenhan Xiong, Xiang Li, Srini Iyer, Jingfei Du, Patrick Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, and Barlas Oguz. 2021. Answering complex open-domain questions with multi-hop dense retrieval. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Wenhan+Xiong%2C+Xiang+Li%2C+Srini+Iyer%2C+Jingfei+Du%2C+Patrick+Lewis%2C+William+Yang+Wang%2C+Yashar+Mehdad%2C+Scott+Yih%2C+Sebastian+Riedel%2C+Douwe+Kiela%2C+and+Barlas+Oguz%2E+2021%2E+Answering+complex+open%2Ddomain+question)
45. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
46. Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. [Factual probing is MASK: Learning vs. learning to recall](https://md.research.elamur.ai/free/en/papers/zhong-2021-factual-probing-is-mask-learning/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017–5033, Online. Association for Co…
47. Table 4: Hyperparameter settings for all compared methods. lr: learning rate, pt_len: prompt length. Rouge (R) BLEU R-1 R-2 R-L PLM-FT PLM-FT (Iter) 74.3 83.6 62.4 76.3 72.7 82.3 52.9 70.8 Prompt-T Prompt-T (Iter) Prefix-T Prefix-T (Iter) 68.7 74.5 70.8 79.0 55.5 64.7 57.8 70.3 66.4 73.7 68.9 77.6 4… [find](https://scholar.google.com/scholar?q=Table+4%3A+Hyperparameter+settings+for+all+compared+methods%2E+lr%3A+learning+rate%2C+pt%5Flen%3A+prompt+length%2E+Rouge+%28R%29+BLEU+R%2D1+R%2D2+R%2DL+PLM%2DFT+PLM%2DFT+%28Iter%29+74%2E3+83%2E6+62%2E4+76%2E3+72%2E7+82%2E3+52%2E9+70%2E8+Prompt%2DT+Pr)

*Updated 2 November 2026*