---
title: "Rationale-Augmented Ensembles in Language Models (2022)"
description: "They then propose rationale-augmented ensembles: many rationale-based predictions are produced… 52 references, cited by 73 papers here."
authors: ["Xuezhi Wang", "Jason Wei", "Dale Schuurmans", "Quoc Le", "Ed Chi", "Denny Zhou"]
year: 2022
venue: "arXiv:2207.00747"
source: "https://arxiv.org/abs/2207.00747"
doi: "10.48550/arxiv.2207.00747"
arxiv: "2207.00747"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2022-rationale-augmented-ensembles-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2022-rationale-augmented-ensembles-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Rationale-Augmented Ensembles in Language Models

# Rationale-Augmented Ensembles in Language Models

**Authors:** [Xuezhi Wang](https://md.research.elamur.ai/free/en/authors/xuezhi-wang/), [Jason Wei](https://md.research.elamur.ai/free/en/authors/jason-wei/), [Dale Schuurmans](https://md.research.elamur.ai/free/en/authors/dale-schuurmans/), [Quoc Le](https://md.research.elamur.ai/free/en/authors/quoc-viet-le/), [Ed Chi](https://md.research.elamur.ai/free/en/authors/ed-chi/) and [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/) · **Year:** 2022 · **Venue:** arXiv:2207.00747 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2207.00747) · [PDF](https://arxiv.org/pdf/2207.00747) · [DOI 10.48550/arxiv.2207.00747](https://doi.org/10.48550/arxiv.2207.00747) · [arXiv 2207.00747](https://arxiv.org/abs/2207.00747) · [OpenAlex](https://openalex.org/W4283828387)

**Citations:** 28 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/wang-2022-rationale-augmented-ensembles-language-models/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Hand-written rationales (chain-of-thought style explanations) in few-shot prompts can improve multi-step reasoning in large language models, but the gains are unreliable. The explanations a prompt author writes may be weak, and they can push accuracy below plain input-to-output prompting. Wang et al. (Google Research) measure this sensitivity on PaLM-540B. They then propose rationale-augmented ensembles: many rationale-based predictions are produced and their answers are combined by voting. Diversity can come from the input side, either by reordering exemplars or by swapping human rationales for model-written ones. It can also come from the output side, by drawing several reasoning paths from the decoder, as in self-consistency. The main finding is that sampling rationales during decoding is what produces robust improvements, regardless of how the prompt is varied. The ensembles outperform both standard few-shot prompting and chain-of-thought prompting across many tasks. This includes tasks that do not obviously need intermediate steps, such as BoolQ, WiC, SST-2 and QQP. The method needs no fine-tuning, verifiers or extra datasets, so it works with any off-the-shelf LLM and provides interpretable explanations alongside its predictions.

## Key points
- **Sensitivity study:** The authors kept the exemplars fixed on e-SNLI, BoolQ, WiC and SST-2 and changed only the rationales.
  - Accuracy varied substantially with this change alone.
  - On e-SNLI, WiC and SST-2, human-written rationales scored worse than prompts with no rationales at all.
  - Model-sampled rationales frequently beat the human-written ones.
- **Rationale generation:** For each held-out exemplar, the authors sampled 1,024 candidate rationales from PaLM-540B. They kept only the candidates whose final answer matched the gold label.
- **Three ensemble variants:**
  - self-consistency, which samples multiple outputs;
  - prompt-order ensembles, which permute the exemplars;
  - input-rationale ensembles, which replace human rationales with generated ones.
  
  Output-space sampling is the component that matters most across all three.
- **Low cost:** The framework needs only a handful of seed rationales (K roughly 3–6 exemplars). It requires no extra training, calibrators, verifiers or auxiliary data.
- **Limitation:** Generated rationales tend to copy the style of the seed rationales, for example the pattern "The first… the second…". The authors therefore recommend writing diverse seeds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,462 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi and Denny Zhou. 2022. Rationale-Augmented Ensembles in Language Models. arXiv:2207.00747.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{wang2022rationale,
  title = {{Rationale-Augmented Ensembles in Language Models}},
  author = {Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi and Denny Zhou},
  year = {2022},
  doi = {10.48550/arxiv.2207.00747},
  eprint = {2207.00747},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2207.00747}
}
```

- [← previous FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/wan-2022-factpegasus-factuality-aware-pre-training/)
- [next → Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark](https://md.research.elamur.ai/free/en/papers/dziri-2022-evaluating-attribution-dialogue-systems-begin/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

52 references, 37 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ackley, D. H., Hinton, G. E., and Sejnowski, T. J. A learning algorithm for boltzmann machines. Cognitive Science, 9(1):147–169, 1985. ISSN 0364-0213. URL [https://www.sciencedirect.com/science/article/pii/S0364021385800124](https://www.sciencedirect.com/science/article/pii/S0364021385800124). [↗](https://www.sciencedirect.com/science/article/pii/S0364021385800124)
2. Asai, A., Hashimoto, K., Hajishirzi, H., Socher, R., and Xiong, C. [Learning to retrieve reasoning paths over wikipedia graph for question answering](https://md.research.elamur.ai/free/en/papers/asai-2019-learning-retrieve-reasoning-paths-over/). In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=SJgVHkrYDH](https://openreview.net/forum?id=SJgVHkrYDH). [↗](https://openreview.net/forum?id=SJgVHkrYDH)
3. Bach, S. H., Sanh, V., Yong, Z.-X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., Alyafeai, Z., Dey, M., Santilli, A., Sun, Z., Ben-David, S., Xu, C., Chhablani, G., Wang, H., Fries, J. A., Al-shaibani, M. S., Sharma, S., Thakker, U., Almubarak, K., Tang, X., Tan… — [PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts](https://md.research.elamur.ai/free/en/papers/bach-2022-promptsource-integrated-development-environment-repository/)
4. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., Giampiccolo, D., Magnini, B., and Szpektor, I. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL challenges workshop on recognising textual entailment, 2006. [find](https://scholar.google.com/scholar?q=Bar%2DHaim%2C+R%2E%2C+Dagan%2C+I%2E%2C+Dolan%2C+B%2E%2C+Ferro%2C+L%2E%2C+Giampiccolo%2C+D%2E%2C+Magnini%2C+B%2E%2C+and+Szpektor%2C+I%2E+The+second+pascal+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+second+PASCAL+challenges)
5. Bentivogli, L., Clark, P., Dagan, I., and Giampiccolo, D. The fifth pascal recognizing textual entailment challenge. In TAC, 2009. [find](https://scholar.google.com/scholar?q=Bentivogli%2C+L%2E%2C+Clark%2C+P%2E%2C+Dagan%2C+I%2E%2C+and+Giampiccolo%2C+D%2E+The+fifth+pascal+recognizing+textual+entailment+challenge%2E+In+TAC%2C+2009%2E)
6. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
7. Camburu, O.-M., Rocktäschel, T., Lukasiewicz, T., and Blunsom, P. [e-snli: Natural language inference with natural language explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/). In Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31, pp. 9539–9…
8. Chen, J., Lin, S., and Durrett, G. Multi-hop question answering via reasoning chains. CoRR, abs/1910.02610, 2019. URL http://arxiv.org/abs/1910.02610. — [arXiv:1910.02610](https://arxiv.org/abs/1910.02610) [↗](http://arxiv.org/abs/1910.02610)
9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
10. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., Schuh, P., Shi, K., Tsvyashchenko, S., Maynez, J., Rao, A., Barnes, P., Tay, Y., Shazeer, N., Prabhakaran, V., Reif, E., Du, N., Hutchinson, B., Pope, R., Bradbury, J., Aust… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
11. Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In NAACL, 2019.
12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). ArXiv, abs/1803.05457, 2018.
13. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/), 2021.
14. Creswell, A., Shanahan, M., and Higgins, I. Selection-inference: Exploiting large language models for interpretable logical reasoning, 2022. URL https://arxiv.org/abs/2205.09712. [↗](https://arxiv.org/abs/2205.09712)
15. Dagan, I., Glickman, O., and Magnini, B. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, pp. 177–190. Springer, 2005. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%2C+Glickman%2C+O%2E%2C+and+Magnini%2C+B%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+pp%2E+177%E2%80%93190%2E+Springer%2C+2005%2E)
16. Ficler, J. and Goldberg, Y. [Controlling linguistic style aspects in neural language generation](https://md.research.elamur.ai/free/en/papers/ficler-2017-controlling-linguistic-style-aspects-neural/). In Proceedings of the Workshop on Stylistic Variation, pp. 94–104, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4912. URL https://aclanthology.org/… [↗](https://aclanthology.org/…)
17. Gao, T., Fisch, A., and Chen, D. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816–3830,…
18. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. [The third pascal recognizing textual entailment challenge](https://md.research.elamur.ai/free/en/papers/giampiccolo-2007-third-pascal-recognizing-textual-entailment/). In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1–9. Association for Computational Linguistics, 2007.
19. Iyer, S., Dandekar, N., and Csernai, K. First quora dataset release: Question pairs, 2017. URL [https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs). [↗](https://data.quora.com/First-Quora-Dataset-Release-Question-Pairs)
20. Jung, J., Qin, L., Welleck, S., Brahman, F., Bhagavatula, C., Bras, R. L., and Choi, Y. [Maieutic prompting: Logically consistent reasoning with recursive explanations](https://md.research.elamur.ai/free/en/papers/jung-2022-maieutic-prompting-logically-consistent-reasoning/), 2022. URL https://arxiv.org/abs/2205.11822. [↗](https://arxiv.org/abs/2205.11822)
21. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/), 2022. URL https://arxiv.org/abs/2205.11916. [↗](https://arxiv.org/abs/2205.11916)
22. Lampinen, A. K., Dasgupta, I., Chan, S. C. Y., Matthewson, K., Tessler, M. H., Creswell, A., McClelland, J. L., Wang, J. X., and Hill, F. [Can language models learn from explanations in context?](https://md.research.elamur.ai/free/en/papers/lampinen-2022-can-language-models-learn-explanations/), 2022. URL https://arxiv.org/abs/2204.02329. [↗](https://arxiv.org/abs/2204.02329)
23. Lester, B., Al-Rfou, R., and Constant, N. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045–3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Lingui…
24. Li, Y., Lin, Z., Zhang, S., Fu, Q., Chen, B., Lou, J.-G., and Chen, W. On the advance of making language models better reasoners, 2022. URL https://arxiv.org/abs/2206.02336. — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [↗](https://arxiv.org/abs/2206.02336)
25. Ling, W., Yogatama, D., Dyer, C., and Blunsom, P. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017. doi: 10.18653/v1/P17-1015. URL… [find](https://scholar.google.com/scholar?q=Ling%2C+W%2E%2C+Yogatama%2C+D%2E%2C+Dyer%2C+C%2E%2C+and+Blunsom%2C+P%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Ass)
26. Liu, J., Liu, A., Lu, X., Welleck, S., West, P., Le Bras, R., Choi, Y., and Hajishirzi, H. Generated knowledge prompting for commonsense reasoning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3154–3169, Dublin, Ireland, May… [find](https://scholar.google.com/scholar?q=Liu%2C+J%2E%2C+Liu%2C+A%2E%2C+Lu%2C+X%2E%2C+Welleck%2C+S%2E%2C+West%2C+P%2E%2C+Le+Bras%2C+R%2E%2C+Choi%2C+Y%2E%2C+and+Hajishirzi%2C+H%2E+Generated+knowledge+prompting+for+commonsense+reasoning%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Ass)
27. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. [What makes good in-context examples for GPT-3?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100– 114, Dublin, Ireland and Online…
28. Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stenetorp, P. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). ArXiv, abs/2104.08786, 2021.
29. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, Brussels, Belgium, October-November 2018. Associatio… [find](https://scholar.google.com/scholar?q=Mihaylov%2C+T%2E%2C+Clark%2C+P%2E%2C+Khot%2C+T%2E%2C+and+Sabharwal%2C+A%2E+Can+a+suit+of+armor+conduct+electricity%3F+a+new+dataset+for+open+book+question+answering%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods)
30. Narang, S., Raffel, C., Lee, K., Roberts, A., Fiedel, N., and Malkan, K. [Wt5?! training text-to-text models to explain their predictions](https://md.research.elamur.ai/free/en/papers/narang-2020-wt5-training-text-text-models/), 2020. URL https://arxiv.org/abs/2004.14546. [↗](https://arxiv.org/abs/2004.14546)
31. Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., and Kiela, D. [Adversarial NLI: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020.
32. Nye, M., Andreassen, A. J., Gur-Ari, G., Michalewski, H., Austin, J., Bieber, D., Dohan, D., Lewkowycz, A., Bosma, M., Luan, D., Sutton, C., and Odena, A. [Show your work: Scratchpads for intermediate computation with language models](https://md.research.elamur.ai/free/en/papers/al-2021-show-your-work-scratchpads-intermediate/), 2021. URL https://arxiv.org/abs/2112. 00114. [↗](https://arxiv.org/abs/2112)
33. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
34. Pi, X., Liu, Q., Chen, B., Ziyadi, M., Lin, Z., Gao, Y., Fu, Q., Lou, J.-G., and Chen, W. Reasoning like program executors, 2022. [find](https://scholar.google.com/scholar?q=Pi%2C+X%2E%2C+Liu%2C+Q%2E%2C+Chen%2C+B%2E%2C+Ziyadi%2C+M%2E%2C+Lin%2C+Z%2E%2C+Gao%2C+Y%2E%2C+Fu%2C+Q%2E%2C+Lou%2C+J%2E%2DG%2E%2C+and+Chen%2C+W%2E+Reasoning+like+program+executors%2C+2022%2E)
35. Pilehvar, M. T. and Camacho-Collados, J. [WiC: the word-in-context dataset for evaluating context-sensitive meaning representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Sh…
36. Rajani, N. F., McCann, B., Xiong, C., and Socher, R. [Explain yourself! Leveraging language models for commonsense reasoning](https://md.research.elamur.ai/free/en/papers/rajani-2019-explain-yourself-leveraging-language-models/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. doi: 10.18653/v1/P19-1487. URL https://aclanthology.org/P19-1487. [↗](https://aclanthology.org/P19-1487)
37. Roberts, A., Raffel, C., and Shazeer, N. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418–5426, Online, November 2020. Association for Computational Linguistics. doi: 1… [find](https://scholar.google.com/scholar?q=Roberts%2C+A%2E%2C+Raffel%2C+C%2E%2C+and+Shazeer%2C+N%2E+How+much+knowledge+can+you+pack+into+the+parameters+of+a+language+model%3F+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Process)
38. Rubin, O., Herzig, J., and Berant, J. [Learning to retrieve prompts for in-context learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/), 2021. URL https://arxiv.org/abs/2112.08633. [↗](https://arxiv.org/abs/2112.08633)
39. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. [Recursive deep models for semantic compositionality over a sentiment treebank](https://md.research.elamur.ai/free/en/papers/socher-2013-recursive-deep-models-semantic-compositionality/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631–1642, Seattle, Washington, USA, Octo…
40. Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/2201.08239. [↗](https://arxiv.org/abs/2201.08239)
41. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355, Brussels, Belgium, Novem…
42. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Curran Associates Inc., Red Hook, NY, USA, 2019.
43. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/), 2022. URL https://arxiv.org/abs/2203.11171. [↗](https://arxiv.org/abs/2203.11171)
44. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.[11903, 2022](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). URL [https://arxiv.org/pdf/2201.11903](https://arxiv.org/pdf/2201.11903). [↗](https://arxiv.org/pdf/2201.11903)
45. Wiegreffe, S., Hessel, J., Swayamdipta, S., Riedl, M., and Choi, Y. [Reframing human-ai collaboration for generating free-text explanations](https://md.research.elamur.ai/free/en/papers/wiegreffe-2021-reframing-human-ai-collaboration-generating/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online and Se…
46. Williams, A., Nangia, N., and Bowman, S. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112…
47. Xu, W., Deng, Y., Zhang, H., Cai, D., and Lam, W. Exploiting reasoning chains for multi-hop science question answering. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1143–1156, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. U… [find](https://scholar.google.com/scholar?q=Xu%2C+W%2E%2C+Deng%2C+Y%2E%2C+Zhang%2C+H%2E%2C+Cai%2C+D%2E%2C+and+Lam%2C+W%2E+Exploiting+reasoning+chains+for+multi%2Dhop+science+question+answering%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2021%2C+pp%2E+11)
48. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., and Manning, C. D. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2369–2380, Brussels, Belgium, October-No…
49. Ye, X. and Durrett, G. The unreliability of explanations in few-shot in-context learning, 2022. [find](https://scholar.google.com/scholar?q=Ye%2C+X%2E+and+Durrett%2C+G%2E+The+unreliability+of+explanations+in+few%2Dshot+in%2Dcontext+learning%2C+2022%2E)
50. Zaidan, O., Eisner, J., and Piatko, C. Using “annotator rationales” to improve machine learning for text categorization. NAACL, 2007. [URL https://aclanthology.org/N07-1033](https://md.research.elamur.ai/free/en/papers/zaidan-2007-annotator-rationales-improve-machine-learning/). [↗](https://aclanthology.org/N07-1033)
51. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. D. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/), 2022. URL https://arxiv.org/abs/2203.14465. [↗](https://arxiv.org/abs/2203.14465)
52. Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. [Calibrate before use: Improving few-shot performance of language models](https://md.research.elamur.ai/free/en/papers/zhao-2021-calibrate-before-use-improving-few/). In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research. PMLR, 20…

*Updated 2 November 2026*