---
title: "Can We Further Elicit Reasoning in LLMs? Critic-Guided"
description: "The authors propose CR-Planner, which treats problem solving as planning over three sub-goal types… 39 references, cited by 11 papers here."
authors: ["Xingxuan Li", "Weiwen Xu", "Ruochen Zhao", "Fangkai Jiao", "Shafiq Joty", "Lidong Bing"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.1244/"
doi: "10.18653/v1/2025.acl-long.1244"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2025-can-we-further-elicit-reasoning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2025-can-we-further-elicit-reasoning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/) / Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

# Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

**Authors:** [Xingxuan Li](https://md.research.elamur.ai/free/en/authors/xingxuan-li/), [Weiwen Xu](https://md.research.elamur.ai/free/en/authors/weiwen-xu/), [Ruochen Zhao](https://md.research.elamur.ai/free/en/authors/ruochen-zhao/), [Fangkai Jiao](https://md.research.elamur.ai/free/en/authors/fangkai-jiao/), [Shafiq Joty](https://md.research.elamur.ai/free/en/authors/shafiq-joty/) and [Lidong Bing](https://md.research.elamur.ai/free/en/authors/lidong-bing/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.1244/) · [PDF](https://aclanthology.org/2025.acl-long.1244.pdf) · [DOI 10.18653/v1/2025.acl-long.1244](https://doi.org/10.18653/v1/2025.acl-long.1244) · [OpenAlex](https://openalex.org/W4412889921)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [11](https://md.research.elamur.ai/free/en/papers/li-2025-can-we-further-elicit-reasoning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs that combine chain-of-thought with retrieval-augmented generation often fail on hard, knowledge-heavy problems such as competitive programming. One flawed reasoning step or an irrelevant retrieved document can derail the whole solution. The authors propose CR-Planner, which treats problem solving as planning over three sub-goal types: REASON (writing rationales), GENQUERY (formulating search queries) and RETRIEVE (picking documents). A sub-goal critic decides which type of step to take next. For that step, a large generator (e.g., GPT-4) proposes several candidates, and an execution critic trained for that candidate type picks the best one. The critics are small fine-tuned models (8B Llama-based reward models). They are trained on step-level rewards from trajectories gathered with Monte Carlo Tree Search, which avoids costly human annotation. The method was tested on competitive programming, theorem-based math and difficult domain retrieval, and it beats baselines by about 10 points on average. Because the critics are external, the approach also works with closed-source generators that cannot be fine-tuned.

## Key points
- CR-Planner alternates between Sub-Goal Selection and Execution Selection. Separate critics score sub-goals and the candidate rationales, queries or documents, and the critics are trained per domain.
- MCTS simulates reasoning and retrieval trajectories and back-propagates long-term rewards to label each step for critic training.
- On USACO, accuracy rises from 10.10 with standard prompting to 12.42 with vanilla MCTS at inference and 17.59 with CR-Planner. Applying MCTS directly at inference appears to overfit to visible test cases.
- LoRA fine-tuning of Llama-3.1-70B-Instruct on the same MCTS data beats plain prompting but still trails CR-Planner, which also needs less compute. A single unified critic (Skywork-Reward-Llama-3.1-8B) performs worse than task-specific critics.
- Sampling three candidates per execution step is enough, with gains levelling off after that. The method's limits are expensive MCTS data collection and its inability to recover when none of the sampled candidates is correct.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 56 works · 17,867 citations
- [Alibaba Group](https://md.research.elamur.ai/free/en/companies/alibaba-group/) — Company, Hangzhou, China · 168 works · 15,386 citations
- [Nanyang Technological University](https://md.research.elamur.ai/free/en/universities/nanyang-technological-university/) — University, Singapore, Singapore · 156 works · 25,644 citations
- [Alibaba DAMO Academy](https://md.research.elamur.ai/free/en/companies/alibaba-damo-academy/) — Company, Hangzhou, China · 39 works · 6,378 citations
- [Alibaba Group (Cayman Islands)](https://md.research.elamur.ai/free/en/companies/alibaba-group-cayman-islands/) — Company, George Town, Cayman Islands · 32 works · 3,389 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works

## Cite

Cite as: Xingxuan Li, Weiwen Xu, Ruochen Zhao, Fangkai Jiao, Shafiq Joty and Lidong Bing. 2025. Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2025can,
  title = {{Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks}},
  author = {Xingxuan Li, Weiwen Xu, Ruochen Zhao, Fangkai Jiao, Shafiq Joty and Lidong Bing},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.1244},
  url = {https://doi.org/10.18653/v1/2025.acl-long.1244}
}
```

- [← previous Can a Single Model Master Both Multi-turn Conversations and Tool Use? CoALM: A Unified Conversational Agentic Language Model](https://md.research.elamur.ai/free/en/papers/acikgoz-2025-can-single-model-master-both/)
- [next → Causal Interventions Reveal Shared Structure Across English Filler–Gap Constructions](https://md.research.elamur.ai/free/en/papers/boguraev-2025-causal-interventions-reveal-shared-structure/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/6/)

## References

39 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. [Self-rag: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/). Proceedings of ICLR.
2. Cameron Browne, Edward Powley, Daniel Whitehouse, Simon Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. 2012. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games. [find](https://scholar.google.com/scholar?q=Cameron+Browne%2C+Edward+Powley%2C+Daniel+Whitehouse%2C+Simon+Lucas%2C+Peter+Cowling%2C+Philipp+Rohlfshagen%2C+Stephen+Tavener%2C+Diego+Perez%2C+Spyridon+Samothrakis%2C+and+Simon+Colton%2E+2012%2E+A+survey+of+monte+carlo+t)
3. Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. [Learning to rank using gradient descent](https://md.research.elamur.ai/free/en/papers/burges-2005-learning-rank-using-gradient/). Proceedings of ICML.
4. Hailin Chen, Fangkai Jiao, Xingxuan Li, Chengwei Qin, Mathieu Ravaut, Ruochen Zhao, Caiming Xiong, and Shafiq Joty. 2024. Chatgpt’s one-year anniversary: Are open-source large language models catching up? arXiv preprint arXiv:2311.16989. — [arXiv:2311.16989](https://arxiv.org/abs/2311.16989) [find](https://scholar.google.com/scholar?q=Hailin+Chen%2C+Fangkai+Jiao%2C+Xingxuan+Li%2C+Chengwei+Qin%2C+Mathieu+Ravaut%2C+Ruochen+Zhao%2C+Caiming+Xiong%2C+and+Shafiq+Joty%2E+2024%2E+Chatgpt%E2%80%99s+one%2Dyear+anniversary%3A+Are+open%2Dsource+large+language+models+catching)
5. Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. 2023. [Theoremqa: A theorem-driven question answering dataset](https://md.research.elamur.ai/free/en/papers/chen-2023-theoremqa-theorem-driven-question-answering/). Proceedings of EMNLP.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. [Critic: Large language models can self-correct with tool-interactive critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/). Proceedings of ICLR.
8. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874.
9. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/). arXiv preprint arXiv:2106.09685.
10. Jie Huang and Kevin Chen-Chuan Chang. 2023. [Towards reasoning in large language models: A survey](https://md.research.elamur.ai/free/en/papers/huang-2023-reasoning-large-language-models-survey/). Proceedings of ACL.
11. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. [A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions](https://md.research.elamur.ai/free/en/papers/huang-2023-survey-hallucination-large-language-models/). arXiv preprint arXiv:2311.05232.
12. Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, Yikai Zhang, Yuqing Yang, Ting Wu, Binjie Wang, Shichao Sun, Yang Xiao, Yiyuan Li, Fan Zhou, Steffi Chern, Yiwei Qin, Yan Ma, Jiadi Su, Yixiu Liu, Yuxiang Zheng, Shaoting Zhang… [find](https://scholar.google.com/scholar?q=Zhen+Huang%2C+Zengzhi+Wang%2C+Shijie+Xia%2C+Xuefeng+Li%2C+Haoyang+Zou%2C+Ruijie+Xu%2C+Run%2DZe+Fan%2C+Lyumanshan+Ye%2C+Ethan+Chern%2C+Yixin+Ye%2C+Yikai+Zhang%2C+Yuqing+Yang%2C+Ting+Wu%2C+Binjie+Wang%2C+Shichao+Sun%2C+Yang+Xiao%2C+Yiyu)
13. Shayekh Islam, Md Rahman, K Hossain, Enamul Hoque, Shafiq Joty, and Md Parvez. 2024. Open-RAG: Enhanced Retrieval Augmented Reasoning with Open-Source Large Language Models. Findings of EMNLP. [find](https://scholar.google.com/scholar?q=Shayekh+Islam%2C+Md+Rahman%2C+K+Hossain%2C+Enamul+Hoque%2C+Shafiq+Joty%2C+and+Md+Parvez%2E+2024%2E+Open%2DRAG%3A+Enhanced+Retrieval+Augmented+Reasoning+with+Open%2DSource+Large+Language+Models%2E+Findings+of+EMNLP%2E)
14. Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen, and Shafiq Joty. 2024. [Learning planning-based reasoning by trajectories collection and process reward synthesizing](https://md.research.elamur.ai/free/en/papers/jiao-2024-learning-planning-based-reasoning-trajectories/). Proceedings of EMNLP.
15. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Proceedings of NIPS.
16. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. [Camel: Communicative agents for "mind" exploration of large language model society](https://md.research.elamur.ai/free/en/papers/li-2023-camel-communicative-agents-mind/). arXiv preprint arXiv:2303.17760.
17. Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Shafiq Joty, Soujanya Poria, and Lidong Bing. 2024. Chain-of-knowledge: Grounding large language models via dynamic knowledge adapting over heterogeneous sources. Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Xingxuan+Li%2C+Ruochen+Zhao%2C+Yew+Ken+Chia%2C+Bosheng+Ding%2C+Shafiq+Joty%2C+Soujanya+Poria%2C+and+Lidong+Bing%2E+2024%2E+Chain%2Dof%2Dknowledge%3A+Grounding+large+language+models+via+dynamic+knowledge+adapting+over+heter)
18. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). Proceedings of ICLR.
19. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Dani+Yogatama%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+Program+induction+by+rationale+generation%3A+Learning+to+solve+and+explain+algebraic+word+problems%2E+Proceedings+of+ACL%2E)
20. Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. [Evaluating the logical reasoning ability of chatgpt and gpt-4](https://md.research.elamur.ai/free/en/papers/liu-2023-evaluating-logical-reasoning-ability-chatgpt/). arXiv preprint arXiv:2304.03439.
21. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
22. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
23. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350. [find](https://scholar.google.com/scholar?q=Ofir+Press%2C+Muru+Zhang%2C+Sewon+Min%2C+Ludwig+Schmidt%2C+Noah+A%2E+Smith%2C+and+Mike+Lewis%2E+2023%2E+Measuring+and+narrowing+the+compositionality+gap+in+language+models%2E+arXiv+preprint+arXiv%3A2210%2E03350%2E)
24. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. [Is ChatGPT a general-purpose natural language processing task solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) Proceedings of EMNLP.
25. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). Proceedings of NIPS.
26. Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. 2024. [Can language models solve olympiad programming?](https://md.research.elamur.ai/free/en/papers/shi-2024-language-models-solve-olympiad-programming/) arXiv preprint arXiv:2404.10952.
27. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. 2023. [Replug: Retrieval-augmented black-box language models](https://md.research.elamur.ai/free/en/papers/shi-2023-replug-retrieval-augmented-black-box/). arXiv preprint arXiv:2301.12652.
28. Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. [Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/). arXiv preprint arXiv:2303.11366.
29. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. Proceedings of NIPS. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Thomas+L%2E+Griffiths%2C+Yuan+Cao%2C+and+Karthik+R+Narasimhan%2E+2023a%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+Proceedings+o)
30. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In Proceedings of ICLR.
31. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). arXiv preprint arXiv:2308.01825.
32. Chong Zhang, Yue Deng, Xiang Lin, Bin Wang, Dianwen Ng, Hai Ye, Xingxuan Li, Yao Xiao, Zhanfeng Mo, Qi Zhang, and Lidong Bing. 2025. [100 days after deepseek-r1: A survey on replication studies and more directions for reasoning language models](https://md.research.elamur.ai/free/en/papers/zhang-2025-100-days-after-deepseek/). arXiv preprint arXiv:2505.00551.
33. Ruochen Zhao, Xingxuan Li, Yew Ken Chia, Bosheng Ding, and Lidong Bing. 2023a. Can chatgpt-like generative models guarantee factual accuracy? on the mistakes of new generation search engines. arXiv preprint arXiv:2304.11076. — [arXiv:2304.11076](https://arxiv.org/abs/2304.11076) [find](https://scholar.google.com/scholar?q=Ruochen+Zhao%2C+Xingxuan+Li%2C+Yew+Ken+Chia%2C+Bosheng+Ding%2C+and+Lidong+Bing%2E+2023a%2E+Can+chatgpt%2Dlike+generative+models+guarantee+factual+accuracy%3F+on+the+mistakes+of+new+generation+search+engines%2E+arXiv+pr)
34. Ruochen Zhao, Xingxuan Li, Shafiq Joty, Chengwei Qin, and Lidong Bing. 2023b. [Verify-and-edit: A knowledge-enhanced chain-of-thought framework](https://md.research.elamur.ai/free/en/papers/zhao-2023-verify-edit-knowledge-enhanced-chain/). Proceedings of ACL.
35. Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Deli Zhao, and Lidong Bing. 2024. Auto arena of llms: Automating llm evaluations with agent peer-battles and committee discussions. arXiv preprint arXiv:2405.20267. — [arXiv:2405.20267](https://arxiv.org/abs/2405.20267) [find](https://scholar.google.com/scholar?q=Ruochen+Zhao%2C+Wenxuan+Zhang%2C+Yew+Ken+Chia%2C+Deli+Zhao%2C+and+Lidong+Bing%2E+2024%2E+Auto+arena+of+llms%3A+Automating+llm+evaluations+with+agent+peer%2Dbattles+and+committee+discussions%2E+arXiv+preprint+arXiv%3A2405)
36. Skywork. 2024. Skywork reward model series. HuggingFace. [find](https://scholar.google.com/scholar?q=Skywork%2E+2024%2E+Skywork+reward+model+series%2E+HuggingFace%2E)
37. Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, and Tao Yu. 2024. [Bright: A realistic and challenging benchmark for reasoning-intensive retrieval](https://md.research.elamur.ai/free/en/papers/su-2024-bright-realistic-challenging-benchmark-reasoning/). arXiv prepr…
38. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Proceedings of NIPS.
39. Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. [Corrective Retrieval Augmented Generation](https://md.research.elamur.ai/free/en/papers/yan-2024-corrective-retrieval-augmented-generation/). arXiv preprint arXiv:2401.15884.

*Updated 31 October 2026*