---
title: "Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement"
description: "Letting an LLM critique and revise its own output helps on some tasks but hurts on others. 43 references, cited by 56 papers here."
authors: ["Wenda Xu", "Guanglei Zhu", "Xuandong Zhao", "Liangming Pan", "Lei Li", "William Wang"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.acl-long.826/"
doi: "10.18653/v1/2024.acl-long.826"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2024-pride-prejudice-llm-amplifies-self/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2024-pride-prejudice-llm-amplifies-self/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement

# Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement

**Authors:** [Wenda Xu](https://md.research.elamur.ai/free/en/authors/wenda-xu/), Guanglei Zhu, Xuandong Zhao, [Liangming Pan](https://md.research.elamur.ai/free/en/authors/liangming-pan/), [Lei Li](https://md.research.elamur.ai/free/en/authors/lei-li/) and [William Wang](https://md.research.elamur.ai/free/en/authors/william-wang/) · **Year:** 2024 · **Venue:** [Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2024](https://md.research.elamur.ai/free/en/venues/acl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-long.826/) · [PDF](https://aclanthology.org/2024.acl-long.826.pdf) · [DOI 10.18653/v1/2024.acl-long.826](https://doi.org/10.18653/v1/2024.acl-long.826) · [OpenAlex](https://openalex.org/W4402671190)

**Citations:** 30 citations (OpenAlex, 2026-10-14) · [56](https://md.research.elamur.ai/free/en/papers/xu-2024-pride-prejudice-llm-amplifies-self/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Letting an LLM critique and revise its own output helps on some tasks but hurts on others. The authors attribute this inconsistency to self-bias: models rate their own generations more generously than an independent judge would. They formalise self-bias with two statistics. The first is the average gap between the model's self-assigned quality scores and true quality. The second is distance skewness, which detects an asymmetric error distribution even when the mean gap is zero. Six models (GPT-4, GPT-3.5-Turbo, Gemini, LLaMA2, Mixtral, DeepSeekMoE) are evaluated on low-resource translation, constrained text generation and mathematical reasoning. Self-bias appears in every model and task, and it grows as refinement iterates. Revised outputs become more fluent and readable but not better on the targeted criterion, so the loop pursues false-positive gains and drifts toward the model's own style. Two remedies help: larger models, and accurate external feedback, which yields real quality improvements. The results caution against trusting LLM self-evaluation or self-rewarding, including in translation for under-resourced languages.

## Key points
- Self-bias is measured with two complementary meta-metrics: statistical bias (mean of predicted minus true quality) and distance skewness (Szekely and Móri, 2006). Distance skewness ranges from 0 (symmetric) to 1 (entirely one-sided).
- Translation experiments use Flores-200 with Yor-En, Jav-En, Arm-En and Ig-En (100 test sentences each). Feedback follows MQM-style error annotation (−1 minor, −5 major, scores from 0 to −25), and BLEURT serves as a proxy for human judgement.
- In the self-refine loop, a revision is kept only if the model's own feedback score improves. This design lets the bias accumulate across iterations.
- Scaling LLaMA2 on Yor-En shows that the 70B model's self-bias levels off after about the 5th iteration, whereas the 7B and 13B models keep amplifying it.
- External feedback with accurate assessment lowers bias for all tested LLMs and yields consistent BLEURT gains across iterations. Code and data are available at github.com/xu1998hz/llm_self_bias.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [University of California, Santa Barbara](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-barbara/) — University, Santa Barbara, United States · 35 works · 9,287 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li and William Wang. 2024. Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2024pride,
  title = {{Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement}},
  author = {Wenda Xu, Guanglei Zhu, Xuandong Zhao, Liangming Pan, Lei Li and William Wang},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.826},
  url = {https://doi.org/10.18653/v1/2024.acl-long.826}
}
```

- [← previous Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models](https://md.research.elamur.ai/free/en/papers/shi-2024-math-llava-bootstrapping-mathematical-reasoning/)
- [next → The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contexts](https://md.research.elamur.ai/free/en/papers/shen-2024-language-barrier-dissecting-safety-challenges/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

43 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Alex J. Cannon, Stephen R. Sobie, and Trevor Q. Murdock. 2015. Bias correction of gcm precipitation by quantile mapping: How well do methods preserve changes in quantiles and extremes? Journal of Climate, 28(17):6938 – 6959. [find](https://scholar.google.com/scholar?q=Alex+J%2E+Cannon%2C+Stephen+R%2E+Sobie%2C+and+Trevor+Q%2E+Murdock%2E+2015%2E+Bias+correction+of+gcm+precipitation+by+quantile+mapping%3A+How+well+do+methods+preserve+changes+in+quantiles+and+extremes%3F+Journal+of+Clim)
3. Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. [Speak, memory: An archaeology of books known to ChatGPT/GPT-4](https://md.research.elamur.ai/free/en/papers/chang-2023-speak-memory-archaeology-books-known/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312–7327, Singapore. Association for Computational Linguistics.
4. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching large language models to self-debug. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2024%2E+Teaching+large+language+models+to+self%2Ddebug%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
5. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. [find](https://scholar.google.com/scholar?q=Marta+R+Costa%2Djuss%C3%A0%2C+James+Cross%2C+Onur+%C3%87elebi%2C+Maha+Elbayad%2C+Kenneth+Heafield%2C+Kevin+Heffernan%2C+Elahe+Kalbassi%2C+Janice+Lam%2C+Daniel+Licht%2C+Jean+Maillard%2C+et+al%2E+2022%2E+No+language+left+behind%3A+Scaling+h)
6. Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie, Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. 2024. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language mo… [find](https://scholar.google.com/scholar?q=Damai+Dai%2C+Chengqi+Deng%2C+Chenggang+Zhao%2C+R%2E+X%2E+Xu%2C+Huazuo+Gao%2C+Deli+Chen%2C+Jiashi+Li%2C+Wangding+Zeng%2C+Xingkai+Yu%2C+Y%2E+Wu%2C+Zhenda+Xie%2C+Y%2E+K%2E+Li%2C+Panpan+Huang%2C+Fuli+Luo%2C+Chong+Ruan%2C+Zhifang+Sui%2C+and+Wenfen)
7. Daniel Deutsch, Rotem Dror, and Dan Roth. 2022. On the limitations of reference-free evaluations of generated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10960–10977, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics… [find](https://scholar.google.com/scholar?q=Daniel+Deutsch%2C+Rotem+Dror%2C+and+Dan+Roth%2E+2022%2E+On+the+limitations+of+reference%2Dfree+evaluations+of+generated+text%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
8. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. [Experts, errors, and context: A large-scale study of human evaluation for machine translation](https://md.research.elamur.ai/free/en/papers/freitag-2021-experts-errors-context-large-scale/). Transactions of the Association for Computational Linguistics, 9:1460– 1474.
9. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. [Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust](https://md.research.elamur.ai/free/en/papers/freitag-2022-results-wmt22-metrics-shared-task/). In Proceedings of the Seventh Conferen…
10. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. [find](https://scholar.google.com/scholar?q=Jinlan+Fu%2C+See%2DKiong+Ng%2C+Zhengbao+Jiang%2C+and+Pengfei+Liu%2E+2023%2E+Gptscore%3A+Evaluate+as+you+desire%2E)
11. Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. 2023. [Reinforced self-training (rest) for language modeling](https://md.research.elamur.ai/free/en/papers/gulcehre-2023-reinforced-self-training-rest-language/).
12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). NeurIPS.
13. Ruixin Hong, Hongming Zhang, Xinyu Pang, Dong Yu, and Changshui Zhang. 2023. A closer look at the self-verification abilities of large language models in logical reasoning. CoRR, abs/2311.07954. [find](https://scholar.google.com/scholar?q=Ruixin+Hong%2C+Hongming+Zhang%2C+Xinyu+Pang%2C+Dong+Yu%2C+and+Changshui+Zhang%2E+2023%2E+A+closer+look+at+the+self%2Dverification+abilities+of+large+language+models+in+logical+reasoning%2E+CoRR%2C+abs%2F2311%2E07954%2E)
14. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023a. [Large language models can self-improve](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-can-self/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1051–1068, Singapore. Association for Computational Linguistics.
15. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023b. [Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/). CoRR, abs/2310.01798.
16. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guill)
17. Pei Ke, Bosi Wen, Zhuoer Feng, Xiao Liu, Xuanyu Lei, Jiale Cheng, Shengyuan Wang, Aohan Zeng, Yuxiao Dong, Hongning Wang, Jie Tang, and Minlie Huang. 2023. [Critiquellm: Scaling llm-as-critic for effective and explainable evaluation of large language model generation](https://md.research.elamur.ai/free/en/papers/ke-2023-critiquellm-informative-critique-generation/). CoRR, abs/2311.18702.
18. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin… [find](https://scholar.google.com/scholar?q=Tom+Kocmi%2C+Eleftherios+Avramidis%2C+Rachel+Bawden%2C+Ondr%CC%8Cej+Bojar%2C+Anton+Dvorkovich%2C+Christian+Federmann%2C+Mark+Fishel%2C+Markus+Freitag%2C+Thamme+Gowda%2C+Roman+Grundkiewicz%2C+Barry+Haddow%2C+Philipp+Koehn%2C+Benja)
19. Tom Kocmi and Christian Federmann. 2023. [GEMBA-MQM: Detecting translation quality error spans with GPT-4](https://md.research.elamur.ai/free/en/papers/kocmi-2023-gemba-mqm-detecting-translation-quality/). In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics.
20. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2023. Benchmarking cognitive biases in large language models as evaluators. [find](https://scholar.google.com/scholar?q=Ryan+Koo%2C+Minhwa+Lee%2C+Vipul+Raheja%2C+Jong+Inn+Park%2C+Zae+Myung+Kim%2C+and+Dongyeop+Kang%2E+2023%2E+Benchmarking+cognitive+biases+in+large+language+models+as+evaluators%2E)
21. Sneha Kudugunta, Isaac Rayburn Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. 2023. [MADLAD-400: A multilingual and document-level large audited dataset](https://md.research.elamur.ai/free/en/papers/kudugunta-2023-madlad-400-multilingual-document-level/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Be…
22. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840, Online. As… [find](https://scholar.google.com/scholar?q=Bill+Yuchen+Lin%2C+Wangchunshu+Zhou%2C+Ming+Shen%2C+Pei+Zhou%2C+Chandra+Bhagavatula%2C+Yejin+Choi%2C+and+Xiang+Ren%2E+2020%2E+CommonGen%3A+A+constrained+text+generation+challenge+for+generative+commonsense+reasoning%2E+I)
23. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023a. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Yang+Liu%2C+Dan+Iter%2C+Yichong+Xu%2C+Shuohang+Wang%2C+Ruochen+Xu%2C+and+Chenguang+Zhu%2E+2023a%2E+G%2Deval%3A+NLG+evaluation+using+gpt%2D4+with+better+human+alignment%2E+In+Proceedings+of+the+2023+Conference+on+Empirical+)
24. Yiqi Liu, Nafise Sadat Moosavi, and Chenghua Lin. 2023b. Llms as narcissistic evaluators: When ego inflates evaluation scores. [find](https://scholar.google.com/scholar?q=Yiqi+Liu%2C+Nafise+Sadat+Moosavi%2C+and+Chenghua+Lin%2E+2023b%2E+Llms+as+narcissistic+evaluators%3A+When+ego+inflates+evaluation+scores%2E)
25. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. 2024. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Prakhar+Gupta%2C+Skyler+Hallinan%2C+Luyu+Gao%2C+Sarah+Wiegreffe%2C+Uri+Alon%2C+Nouha+Dziri%2C+Shrimai+Prabhumoye%2C+Yiming+Yang%2C+et+al%2E+2024%2E+Self%2Drefine%3A+Iterative+refinement+with+self%2Df)
26. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9004–9017, Singapore. Association for Computation… [find](https://scholar.google.com/scholar?q=Potsawee+Manakul%2C+Adian+Liusie%2C+and+Mark+Gales%2E+2023%2E+SelfCheckGPT%3A+Zero%2Dresource+black%2Dbox+hallucination+detection+for+generative+large+language+models%2E+In+Proceedings+of+the+2023+Conference+on+Empir)
27. Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2023. Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies. CoRR, abs/2308.03188. [find](https://scholar.google.com/scholar?q=Liangming+Pan%2C+Michael+Saxon%2C+Wenda+Xu%2C+Deepak+Nathani%2C+Xinyi+Wang%2C+and+William+Yang+Wang%2E+2023%2E+Automatically+correcting+large+language+models%3A+Surveying+the+landscape+of+diverse+self%2Dcorrection+stra)
28. Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. [Learning compact metrics for mt](https://md.research.elamur.ai/free/en/papers/pu-2021-learning-compact-metrics-mt/). In Proceedings of EMNLP.
29. Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text generation. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Thibault+Sellam%2C+Dipanjan+Das%2C+and+Ankur+P+Parikh%2E+2020%2E+Bleurt%3A+Learning+robust+metrics+for+text+generation%2E+In+Proceedings+of+ACL%2E)
30. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2024. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). Advances in Neural Information Processing Systems, 36.
31. Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023. GPT-4 doesn’t know it’s wrong: An analysis of iterative prompting for reasoning problems. CoRR, abs/2310.12397. — [arXiv:2310.12397](https://arxiv.org/abs/2310.12397) [find](https://scholar.google.com/scholar?q=Kaya+Stechly%2C+Matthew+Marquez%2C+and+Subbarao+Kambhampati%2E+2023%2E+GPT%2D4+doesn%E2%80%99t+know+it%E2%80%99s+wrong%3A+An+analysis+of+iterative+prompting+for+reasoning+problems%2E+CoRR%2C+abs%2F2310%2E12397%2E+%E2%80%94+%5BarXiv%3A2310%2E12397%5D%28http)
32. Gabor Szekely and Tamás Móri. 2006. A characteristic measure of asymmetry and its application for testing diagonal symmetry. COMMUN. STATIST.—THEORY METH., pages 1633–1639. [find](https://scholar.google.com/scholar?q=Gabor+Szekely+and+Tam%C3%A1s+M%C3%B3ri%2E+2006%2E+A+characteristic+measure+of+asymmetry+and+its+application+for+testing+diagonal+symmetry%2E+COMMUN%2E+STATIST%2E%E2%80%94THEORY+METH%2E%2C+pages+1633%E2%80%931639%2E)
33. Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) - Findings. [find](https://scholar.google.com/scholar?q=Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+%28NAACL%29+%2D+Findings%2E)
34. Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Wang, and Lei Li. 2023. [INSTRUCTSCORE: Towards explainable text generation evaluation with automatic feedback](https://md.research.elamur.ai/free/en/papers/xu-2023-instructscore-explainable-text-generation-evaluation/). In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5967–5994, Sin…
35. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. [find](https://scholar.google.com/scholar?q=Weizhe+Yuan%2C+Richard+Yuanzhe+Pang%2C+Kyunghyun+Cho%2C+Xian+Li%2C+Sainbayar+Sukhbaatar%2C+Jing+Xu%2C+and+Jason+Weston%2E+2024%2E+Self%2Drewarding+language+models%2E)
36. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing… [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+Hao+Zhang%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+2023%2E+Judging+LLM%2Das%2Da%2D)
37. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. [Towards a unified multi-dimensional evaluator for text generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2023–2038, Abu Dhabi, U…
38. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805.
39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+et+al%2E+2023%2E+Llama+2%3A+Open+foundation+and+fi)
40. Gladys Tyen, Hassan Mansoor, Peter Chen, Tony Mak, and Victor Carbune. 2023. Llms cannot find reasoning errors, but can correct them! CoRR, abs/2311.08516. — [arXiv:2311.08516](https://arxiv.org/abs/2311.08516) [find](https://scholar.google.com/scholar?q=Gladys+Tyen%2C+Hassan+Mansoor%2C+Peter+Chen%2C+Tony+Mak%2C+and+Victor+Carbune%2E+2023%2E+Llms+cannot+find+reasoning+errors%2C+but+can+correct+them%21+CoRR%2C+abs%2F2311%2E08516%2E+%E2%80%94+%5BarXiv%3A2311%2E08516%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2)
41. Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023. Can large language models really improve by self-critiquing their own plans? CoRR, abs/2310.08118. — [arXiv:2310.08118](https://arxiv.org/abs/2310.08118) [find](https://scholar.google.com/scholar?q=Karthik+Valmeekam%2C+Matthew+Marquez%2C+and+Subbarao+Kambhampati%2E+2023%2E+Can+large+language+models+really+improve+by+self%2Dcritiquing+their+own+plans%3F+CoRR%2C+abs%2F2310%2E08118%2E+%E2%80%94+%5BarXiv%3A2310%2E08118%5D%28https%3A%2F%2Farxi)
42. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
43. Wenda Xu, Daniel Deutsch, Mara Finkelstein, Juraj Juraska, Biao Zhang, Zhongtao Liu, William Yang Wang, Lei Li, and Markus Freitag. 2024. Llmrefine: Pinpointing and refining large language models via fine-grained actionable feedback. In Proceedings of 2024 [find](https://scholar.google.com/scholar?q=Wenda+Xu%2C+Daniel+Deutsch%2C+Mara+Finkelstein%2C+Juraj+Juraska%2C+Biao+Zhang%2C+Zhongtao+Liu%2C+William+Yang+Wang%2C+Lei+Li%2C+and+Markus+Freitag%2E+2024%2E+Llmrefine%3A+Pinpointing+and+refining+large+language+models+via+)

*Updated 24 October 2026*