---
title: "Universal Self-Consistency for Large Language Model Generation"
description: "The authors, from Google, propose Universal Self-Consistency (USC). 50 references, cited by 76 papers here."
authors: ["Xinyun Chen", "Renat Aksitov", "Uri Alon", "Jie Ren", "Kefan Xiao", "Pengcheng Yin", "Sushant Prakash", "Charles Sutton", "Xuezhi Wang", "Denny Zhou"]
year: 2023
venue: "arXiv:2311.17311"
source: "https://arxiv.org/abs/2311.17311"
doi: "10.48550/arxiv.2311.17311"
arxiv: "2311.17311"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chen-2023-universal-self-consistency-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chen-2023-universal-self-consistency-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / Universal Self-Consistency for Large Language Model Generation

# Universal Self-Consistency for Large Language Model Generation

**Authors:** [Xinyun Chen](https://md.research.elamur.ai/free/en/authors/xinyun-chen/), Renat Aksitov, [Uri Alon](https://md.research.elamur.ai/free/en/authors/uri-alon/), [Jie Ren](https://md.research.elamur.ai/free/en/authors/jie-ren/), Kefan Xiao, [Pengcheng Yin](https://md.research.elamur.ai/free/en/authors/pengcheng-yin/), Sushant Prakash, [Charles Sutton](https://md.research.elamur.ai/free/en/authors/charles-sutton/), [Xuezhi Wang](https://md.research.elamur.ai/free/en/authors/xuezhi-wang/) and [Denny Zhou](https://md.research.elamur.ai/free/en/authors/denny-zhou/) · **Year:** 2023 · **Venue:** arXiv:2311.17311 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2311.17311) · [PDF](https://arxiv.org/pdf/2311.17311) · [DOI 10.48550/arxiv.2311.17311](https://doi.org/10.48550/arxiv.2311.17311) · [arXiv 2311.17311](https://arxiv.org/abs/2311.17311) · [OpenAlex](https://openalex.org/W4389217371)

**Citations:** 10 citations (OpenAlex, 2026-10-14) · [76](https://md.research.elamur.ai/free/en/papers/chen-2023-universal-self-consistency-large-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Standard self-consistency samples several chain-of-thought solutions and takes a majority vote over the final answers. That vote only works when answers can be compared by exact match, such as a single number. It cannot handle free-form outputs like summaries or open-ended answers. The authors, from Google, propose Universal Self-Consistency (USC). USC samples several responses, joins them into one prompt, and asks the same LLM to pick the response that agrees most with the others. No hand-written answer-extraction rules or code execution are needed.

They test USC with PaLM 2-L and gpt-3.5-turbo on several task types:
- math (GSM8K, MATH)
- code (BIRD-SQL, ARCADE)
- long-document summarization (GovReport, SummScreen)
- open-ended QA (TruthfulQA)

On math, USC roughly equals regular self-consistency even when output formats vary. On code, it matches execution-based voting without running any programs. On summarization and TruthfulQA, where classic voting cannot be applied, it improves over greedy decoding and random selection. Its choices largely coincide with standard self-consistency where both apply, and it is reported to be robust to the order of candidates. This makes consistency-based selection usable for generation tasks in general.

## Key points
- **Setup:** 8 candidate samples per input. Temperature is 0.6 for PaLM 2-L and 1.0 for gpt-3.5-turbo. All sampling-based methods choose from the same candidate pool.
- **Benchmarks:**
  - GSM8K (8,500 grade-school problems) and MATH (12,500 competition problems)
  - BIRD-SQL (text-to-SQL) and ARCADE (Python in data-science notebooks)
  - GovReport (~7,900-word inputs, ~500-word summaries) and SummScreen (~5,600-word transcripts, ~100-word recaps), both from ZeroSCROLLS
  - TruthfulQA (817 questions)
- **Metrics:**
  - Summarization: ROUGE-1, ROUGE-2, ROUGE-Lsum and BERTScore F1
  - TruthfulQA: GPT-judge for truthfulness and GPT-info for informativeness
- **Code baselines:** USC is compared with execution-based voting, which clusters programs by their outputs. For ARCADE, a fuzzy-matching variant of that voting is also included.
- **Selection prompt:** The prompt asks the model to name the response with the strongest majority consensus. For list-style answers with no exact overlap, the model favors the response whose individual items recur most across candidates.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang and Denny Zhou. 2023. Universal Self-Consistency for Large Language Model Generation. arXiv:2311.17311.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chen2023universal,
  title = {{Universal Self-Consistency for Large Language Model Generation}},
  author = {Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang and Denny Zhou},
  year = {2023},
  doi = {10.48550/arxiv.2311.17311},
  eprint = {2311.17311},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.17311}
}
```

- [← previous PromptNER: Prompting For Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/ashok-2023-promptner-prompting-named-entity-recognition/)
- [next → InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition](https://md.research.elamur.ai/free/en/papers/zhang-2023-internlm-xcomposer-vision-language/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

50 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, et al. [Towards a human-like open-domain chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/). arXiv preprint arXiv:2001.09977, 2020.
2. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/). arXiv preprint arXiv:2305.10403, 2023.
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
4. Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew R Gormley. It’s mbr all the way down: Modern generation techniques through the lens of minimum bayes risk. arXiv preprint arXiv:2310.01387, 2023. [find](https://scholar.google.com/scholar?q=Amanda+Bertsch%2C+Alex+Xie%2C+Graham+Neubig%2C+and+Matthew+R+Gormley%2E+It%E2%80%99s+mbr+all+the+way+down%3A+Modern+generation+techniques+through+the+lens+of+minimum+bayes+risk%2E+arXiv+preprint+arXiv%3A2310%2E01387%2C+2023%2E)
5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. [Sparks of artificial general intelligence: Early experiments with gpt-4](https://md.research.elamur.ai/free/en/papers/bubeck-2023-sparks-artificial-general-intelligence-early/), 2023.
6. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397, 2022a. [find](https://scholar.google.com/scholar?q=Bei+Chen%2C+Fengji+Zhang%2C+Anh+Nguyen%2C+Daoguang+Zan%2C+Zeqi+Lin%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+Codet%3A+Code+generation+with+generated+tests%2E+arXiv+preprint+arXiv%3A2207%2E10397%2C+2022a%2E)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
8. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. [Summscreen: A dataset for abstractive screenplay summarization](https://md.research.elamur.ai/free/en/papers/chen-2022-summscreen-dataset-abstractive-screenplay-summarization/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8602–8615, 2022b.
9. Xinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Chang+Liu%2C+and+Dawn+Song%2E+Execution%2Dguided+neural+program+synthesis%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E)
10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
11. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+Gptscore%3A+Evaluate+as+you+desire%2C+2023%2E)
12. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. [Critic: Large language models can self-correct with tool-interactive critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/). arXiv preprint arXiv:2305.11738, 2023.
13. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
14. Baizhou Huang, Shuai Lu, Weizhu Chen, Xiaojun Wan, and Nan Duan. Enhancing large language models in coding through multi-perspective self-consistency. arXiv preprint arXiv:2309.17272, 2023a. [find](https://scholar.google.com/scholar?q=Baizhou+Huang%2C+Shuai+Lu%2C+Weizhu+Chen%2C+Xiaojun+Wan%2C+and+Nan+Duan%2E+Enhancing+large+language+models+in+coding+through+multi%2Dperspective+self%2Dconsistency%2E+arXiv+preprint+arXiv%3A2309%2E17272%2C+2023a%2E)
15. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). arXiv preprint arXiv:2310.01798, 2023b.
16. Luyang Huang, Shuyang Cao, Nikolaus Parulian, Heng Ji, and Lu Wang. [Efficient attentions for long document summarization](https://md.research.elamur.ai/free/en/papers/huang-2021-efficient-attentions-long-document-summarization/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1419–1436, 2021.
17. Siddhartha Jain, Xiaofei Ma, Anoop Deoras, and Bing Xiang. Self-consistency for open-ended generations. arXiv preprint arXiv:2307.06857, 2023. — [arXiv:2307.06857](https://arxiv.org/abs/2307.06857) [find](https://scholar.google.com/scholar?q=Siddhartha+Jain%2C+Xiaofei+Ma%2C+Anoop+Deoras%2C+and+Bing+Xiang%2E+Self%2Dconsistency+for+open%2Dended+generations%2E+arXiv+preprint+arXiv%3A2307%2E06857%2C+2023%2E+%E2%80%94+%5BarXiv%3A2307%2E06857%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2307%2E06857%29)
18. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. [Solving quantitative reasoning problems with language models](https://md.research.elamur.ai/free/en/papers/lewkowycz-2022-solving-quantitative-reasoning-problems-language/). Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
19. Jinyang Li, Binyuan Hui, Ge Qu, Binhua Li, Jiaxi Yang, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. [Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls](https://md.research.elamur.ai/free/en/papers/li-2023-llm-already-serve-database-interface/). arXiv preprint arXiv:2305.03111, 2023a.
20. Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. [Making language models better reasoners with step-aware verifier](https://md.research.elamur.ai/free/en/papers/li-2023-making-language-models-better-reasoners/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5315–5333, 2023b.
21. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097, 2022.
22. Stephanie Lin, Jacob Hilton, and Owain Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958, 2021.
23. Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv:2205.14334, 2022. [find](https://scholar.google.com/scholar?q=Stephanie+Lin%2C+Jacob+Hilton%2C+and+Owain+Evans%2E+Teaching+models+to+express+their+uncertainty+in+words%2E+arXiv+preprint+arXiv%3A2205%2E14334%2C+2022%2E)
24. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023. [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+G%2Deval%3A+Nlg+evaluation+using+gpt%2D4+with+better+human+alignment%2C+2023%2E)
25. Marcus J Min, Yangruibo Ding, Luca Buratti, Saurabh Pujar, Gail Kaiser, Suman Jana, and Baishakhi Ray. Beyond accuracy: Evaluating self-consistency of code large language models with identity-chain. arXiv preprint arXiv:2310.14053, 2023. — [arXiv:2310.14053](https://arxiv.org/abs/2310.14053) [find](https://scholar.google.com/scholar?q=Marcus+J+Min%2C+Yangruibo+Ding%2C+Luca+Buratti%2C+Saurabh+Pujar%2C+Gail+Kaiser%2C+Suman+Jana%2C+and+Baishakhi+Ray%2E+Beyond+accuracy%3A+Evaluating+self%2Dconsistency+of+code+large+language+models+with+identity%2Dchain%2E+a)
26. Ansong Ni, Srini Iyer, Dragomir Radev, Ves Stoyanov, Wen-tau Yih, Sida I Wang, and Xi Victoria Lin. [Lever: Learning to verify language-to-code generation with execution](https://md.research.elamur.ai/free/en/papers/ni-2023-lever-learning-verify-language-code/). arXiv preprint arXiv:2302.08468, 2023.
27. Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.01928, 2023. [find](https://scholar.google.com/scholar?q=Allen+Z+Ren%2C+Anushri+Dixit%2C+Alexandra+Bodrova%2C+Sumeet+Singh%2C+Stephen+Tu%2C+Noah+Brown%2C+Peng+Xu%2C+Leila+Takayama%2C+Fei+Xia%2C+Jake+Varley%2C+et+al%2E+Robots+that+ask+for+help%3A+Uncertainty+alignment+for+large+lan)
28. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. [Zeroscrolls: A zero-shot benchmark for long text understanding](https://md.research.elamur.ai/free/en/papers/shaham-2023-zeroscrolls-zero-shot-benchmark-long/). arXiv preprint arXiv:2305.14196, 2023.
29. Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. Generate & rank: A multi-task framework for math word problems. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2269–2279, Punta Cana, Dominican Republic, November 2021. Association for… [find](https://scholar.google.com/scholar?q=Jianhao+Shen%2C+Yichun+Yin%2C+Lin+Li%2C+Lifeng+Shang%2C+Xin+Jiang%2C+Ming+Zhang%2C+and+Qun+Liu%2E+Generate+%26+rank%3A+A+multi%2Dtask+framework+for+math+word+problems%2E+In+Findings+of+the+Association+for+Computational+Lin)
30. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022. [find](https://scholar.google.com/scholar?q=Freda+Shi%2C+Daniel+Fried%2C+Marjan+Ghazvininejad%2C+Luke+Zettlemoyer%2C+and+Sida+I%2E+Wang%2E+Natural+language+to+code+translation+with+execution%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Na)
31. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. [Towards expert-level medical question answering with large language models](https://md.research.elamur.ai/free/en/papers/singhal-2023-towards-expert-level-medical-question/). arXiv preprint arXiv:2305.09617, 2023.
32. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. [Lamda: Language models for dialog applications](https://md.research.elamur.ai/free/en/papers/thoppilan-2022-lamda-language-models-dialog-applications/). arXiv preprint arXiv:2201.08239, 2022. URL https://arxiv.org/abs/ 2201.08239. [↗](https://arxiv.org/abs/)
33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+Llama+2%3A+Open+foundation+and+fine%2Dtun)
34. Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. [Is chatgpt a good nlg evaluator? a preliminary study](https://md.research.elamur.ai/free/en/papers/wang-2023-chatgpt-good-nlg-evaluator-preliminary/), 2023a.
35. Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023b. [find](https://scholar.google.com/scholar?q=Peiyi+Wang%2C+Lei+Li%2C+Liang+Chen%2C+Dawei+Zhu%2C+Binghuai+Lin%2C+Yunbo+Cao%2C+Qi+Liu%2C+Tianyu+Liu%2C+and+Zhifang+Sui%2E+Large+language+models+are+not+fair+evaluators%2E+arXiv+preprint+arXiv%3A2305%2E17926%2C+2023b%2E)
36. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations, 2022.
37. [Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. 2022. URL](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/) [https://arxiv.org/pdf/2201.11903](https://arxiv.org/pdf/2201.11903). [↗](https://arxiv.org/pdf/2201.11903)
38. Gwenyth Portillo Wightman, Alexandra DeLucia, and Mark Dredze. Strength in numbers: Estimating confidence of large language models by prompt agreement. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (TrustNLP 2023), pp. 326–362, 2023. [find](https://scholar.google.com/scholar?q=Gwenyth+Portillo+Wightman%2C+Alexandra+DeLucia%2C+and+Mark+Dredze%2E+Strength+in+numbers%3A+Estimating+confidence+of+large+language+models+by+prompt+agreement%2E+In+Proceedings+of+the+3rd+Workshop+on+Trustworth)
39. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). arXiv preprint arXiv:2309.03409, 2023.
40. Pengcheng Yin and Graham Neubig. Reranking for neural semantic parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [find](https://scholar.google.com/scholar?q=Pengcheng+Yin+and+Graham+Neubig%2E+Reranking+for+neural+semantic+parsing%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+2019%2E)
41. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. Natural language to code generation in interactive data science notebooks. In Proceedings of the 61st Annual Meeting… [find](https://scholar.google.com/scholar?q=Pengcheng+Yin%2C+Wen%2DDing+Li%2C+Kefan+Xiao%2C+Abhishek+Rao%2C+Yeming+Wen%2C+Kensen+Shi%2C+Joshua+Howland%2C+Paige+Bailey%2C+Michele+Catasta%2C+Henryk+Michalewski%2C+Oleksandr+Polozov%2C+and+Charles+Sutton%2E+Natural+language)
42. Ori Yoran, Tomer Wolfson, Ben Bogin, Uri Katz, Daniel Deutch, and Jonathan Berant. [Answering questions by meta-reasoning over multiple chains of thought](https://md.research.elamur.ai/free/en/papers/yoran-2023-answering-questions-meta-reasoning-over/). arXiv preprint arXiv:2304.13007, 2023.
43. Murong Yue, Jie Zhao, Min Zhang, Liang Du, and Ziyu Yao. Large language model cascades with mixture of thoughts representations for cost-efficient reasoning. arXiv preprint arXiv:2310.03094, 2023. — [arXiv:2310.03094](https://arxiv.org/abs/2310.03094) [find](https://scholar.google.com/scholar?q=Murong+Yue%2C+Jie+Zhao%2C+Min+Zhang%2C+Liang+Du%2C+and+Ziyu+Yao%2E+Large+language+model+cascades+with+mixture+of+thoughts+representations+for+cost%2Defficient+reasoning%2E+arXiv+preprint+arXiv%3A2310%2E03094%2C+2023%2E+%E2%80%94+%5B)
44. Lu Zeng, Sree Hari Krishnan Parthasarathi, and Dilek Hakkani-Tur. N-best hypotheses reranking for text-to-sql systems. arXiv preprint arXiv:2210.10668, 2022. — [arXiv:2210.10668](https://arxiv.org/abs/2210.10668) [find](https://scholar.google.com/scholar?q=Lu+Zeng%2C+Sree+Hari+Krishnan+Parthasarathi%2C+and+Dilek+Hakkani%2DTur%2E+N%2Dbest+hypotheses+reranking+for+text%2Dto%2Dsql+systems%2E+arXiv+preprint+arXiv%3A2210%2E10668%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E10668%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs)
45. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2019. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+Bertscore%3A+Evaluating+text+generation+with+bert%2E+In+International+Conference+on+Learning+Representations%2C+2019%2E)
46. Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pp. 41832–41846. PMLR, 2023a. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Tao+Yu%2C+Tatsunori+Hashimoto%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Daniel+Fried%2C+and+Sida+Wang%2E+Coder+reviewer+reranking+for+code+generation%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+4183)
47. Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. [Wider and deeper llm networks are fairer llm evaluators](https://md.research.elamur.ai/free/en/papers/zhang-2023-wider-deeper-llm-networks-fairer/). arXiv preprint arXiv:2308.01862, 2023b.
48. Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. Progressive-hint prompting improves reasoning in large language models. arXiv preprint arXiv:2304.09797, 2023a. [find](https://scholar.google.com/scholar?q=Chuanyang+Zheng%2C+Zhengying+Liu%2C+Enze+Xie%2C+Zhenguo+Li%2C+and+Yu+Li%2E+Progressive%2Dhint+prompting+improves+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2304%2E09797%2C+2023a%2E)
49. Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. On large language models’ selection bias in multi-choice questions. arXiv preprint arXiv:2309.03882, 2023b. [find](https://scholar.google.com/scholar?q=Chujie+Zheng%2C+Hao+Zhou%2C+Fandong+Meng%2C+Jie+Zhou%2C+and+Minlie+Huang%2E+On+large+language+models%E2%80%99+selection+bias+in+multi%2Dchoice+questions%2E+arXiv+preprint+arXiv%3A2309%2E03882%2C+2023b%2E)
50. Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. arXiv preprint arXiv:2205.00049, 2022. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Junxian+He%2C+Xuezhe+Ma%2C+Taylor+Berg%2DKirkpatrick%2C+and+Graham+Neubig%2E+Prompt+consistency+for+zero%2Dshot+task+generalization%2E+arXiv+preprint+arXiv%3A2205%2E00049%2C+2022%2E)

*Updated 24 October 2026*