---
title: "Self-Reflection in LLM Agents: Effects on Problem-Solving"
description: "Renze and Guven test whether an agent that analyses its own failed reasoning does better on a… 51 references, cited by 59 papers here."
authors: ["Matthew Renze", "Erhan Guven"]
year: 2024
venue: "arXiv:2405.06682 · 2nd International Conference on Foundation and Large Language Models (FLLM 2024), pp. 476-483"
source: "https://arxiv.org/abs/2405.06682"
doi: "10.1109/fllm63129.2024.10852493"
arxiv: "2405.06682"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/renze-2024-self-reflection-llm-agents-effects/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/renze-2024-self-reflection-llm-agents-effects/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/) / Self-Reflection in LLM Agents: Effects on Problem-Solving Performance

# Self-Reflection in LLM Agents: Effects on Problem-Solving Performance

**Authors:** [Matthew Renze](https://md.research.elamur.ai/free/en/authors/matthew-renze/) and Erhan Guven · **Year:** 2024 · **Venue:** arXiv:2405.06682 · 2nd International Conference on Foundation and Large Language Models (FLLM 2024), pp. 476-483 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1109/fllm63129.2024.10852493) · [PDF](https://arxiv.org/pdf/2405.06682) · [DOI 10.1109/fllm63129.2024.10852493](https://doi.org/10.1109/fllm63129.2024.10852493) · [arXiv 2405.06682](https://arxiv.org/abs/2405.06682) · [OpenAlex](https://openalex.org/W4406892137)

**Citations:** 16 citations (OpenAlex, 2026-10-14) · [59](https://md.research.elamur.ai/free/en/papers/renze-2024-self-reflection-llm-agents-effects/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
LLM agents that reason via chain of thought (CoT) still make logical, arithmetic and factual errors. Renze and Guven test whether an agent that analyses its own failed reasoning does better on a second attempt. They built a 1,000-item multiple-choice exam by sampling 100 questions from each of ten problem sets (from ARC, AGIEval, HellaSwag and MedMCQA), spanning science, math, law, medicine, logic and English. Nine LLMs first answered every item as a non-reflecting Baseline agent. For each miss, the model saw the correct answer and wrote a reflection. Answer labels and option texts were then masked out, and the reflection was inserted into a re-answer prompt. Eight reflection variants were compared, from a bare "you were wrong" notice to full step-by-step solutions. Self-reflection raised accuracy significantly (p \< 0.001) for every model. Information-rich reflections beat terse ones, and simply knowing about the earlier mistake already helped. The study gives practical guidance for building agents that learn from their errors.

## Key points
- Models tested: GPT-4, GPT-3.5 Turbo, Mistral Large, Gemini 1.5 Pro (Preview), Gemini 1.0 Pro, Command R+, Claude 3 Opus, Llama 2 70B Chat and Llama 2 7B Chat. All were run at temperature 0.0 through cloud APIs.
- Agent types:
  - Baseline (control) and Retry
  - Keywords, Advice, Explanation, Instructions and Solution
  - Composite (all six reflection types combined)
  - Unredacted (keeps the answers; serves as an upper bound)
- Explanation, Instructions, Solution and Composite reflections performed best. Even Retry improved results across all LLMs.
- LSAT-AR was the hardest exam and gained the most from reflection, which may skew the aggregate results. Top models scoring above 90% left little room to show gains on other exams.
- Limitations:
  - Only single-step tasks were tested.
  - API and content-filter errors distorted accuracy by up to 2.8% for Gemini 1.0 Pro and Mistral Large.
- Code and data are available at github.com/matthewrenze/self-reflection.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 187 works · 92,194 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Matthew Renze and Erhan Guven. 2024. Self-Reflection in LLM Agents: Effects on Problem-Solving Performance. arXiv:2405.06682 · 2nd International Conference on Foundation and Large Language Models (FLLM 2024), pp. 476-483.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{renze2024self,
  title = {{Self-Reflection in LLM Agents: Effects on Problem-Solving Performance}},
  author = {Matthew Renze and Erhan Guven},
  year = {2024},
  doi = {10.1109/fllm63129.2024.10852493},
  eprint = {2405.06682},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1109/fllm63129.2024.10852493}
}
```

- [← previous Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives](https://md.research.elamur.ai/free/en/papers/zhang-2024-self-contrast-better-reflection-through/)
- [next → MEGAVERSE: Benchmarking Large Language Models Across Languages, Modalities, Models and Tasks](https://md.research.elamur.ai/free/en/papers/ahuja-2024-megaverse-benchmarking-large-language-models/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/4/)

## References

51 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa, “[Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/),” in Advances in Neural Information Processing Systems, vol. 35, 5 2022, pp. 22 199–22 213. [Online]. Available: https://arxiv.org/abs/2205.11916 [↗](https://arxiv.org/abs/2205.11916)
2. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “[Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/),” arXiv, 1 2022. [Online]. Available: https://arxiv.org/abs/2201.11903 [↗](https://arxiv.org/abs/2201.11903)
3. Y. Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “[Large language models are human-level prompt engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/),” The Eleventh International Conference on Learning Representations, 11 2023. [Online]. Available: https://arxiv.org/abs/2211.01910 [↗](https://arxiv.org/abs/2211.01910)
4. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, D. Chen, H. S. Chan, W. Dai, A. Madotto, and P. Fung, “[Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/),” ACM Computing Surveys, vol. 55, 2 2022. [Online]. Available: http://dx.doi.org/10.1145/3571730 [↗](http://dx.doi.org/10.1145/3571730)
5. M. U. Hadi, qasem al tashi, R. Qureshi, A. Shah, amgad muneer, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wu, S. Mirjalili, Q. Al-Tashi, and A. Muneer, “A survey on large language models: Applications, challenges, limitations, and practical usage,” Authorea Preprints, 10 2023. [Online]. Availab… — [doi:10.36227/techrxiv.23589741.v1](https://doi.org/10.36227/techrxiv.23589741.v1) [find](https://scholar.google.com/scholar?q=M%2E+U%2E+Hadi%2C+qasem+al+tashi%2C+R%2E+Qureshi%2C+A%2E+Shah%2C+amgad+muneer%2C+M%2E+Irfan%2C+A%2E+Zafar%2C+M%2E+B%2E+Shaikh%2C+N%2E+Akhtar%2C+J%2E+Wu%2C+S%2E+Mirjalili%2C+Q%2E+Al%2DTashi%2C+and+A%2E+Muneer%2C+%E2%80%9CA+survey+on+large+language+models%3A+Applica)
6. A. Payandeh, D. Pluth, J. Hosier, X. Xiao, and V. K. Gurbani, “How susceptible are llms to logical fallacies?” arXiv, 8 2023. [Online]. Available: https://arxiv.org/abs/2308.09853v1 [↗](https://arxiv.org/abs/2308.09853v1)
7. J. Huang, X. Chen, S. Mishra, H. S. Zheng, A. W. Yu, X. Song, and D. Zhou, “[Large language models cannot self-correct reasoning yet](https://md.research.elamur.ai/free/en/papers/huang-2023-large-language-models-cannot-self/),” arXiv, 10 2023. [Online]. Available: https://arxiv.org/abs/2310.01798 [↗](https://arxiv.org/abs/2310.01798)
8. Z. Ji, T. Yu, Y. Xu, N. Lee, E. Ishii, and P. Fung, “Towards mitigating hallucination in large language models via self-reflection,” arXiv, 10 2023. [Online]. Available: https://arxiv.org/abs/2310.06271 [↗](https://arxiv.org/abs/2310.06271)
9. S. Minaee, T. Mikolov, N. Nikzad, M. Chenaghlu, R. Socher, X. Amatriain, and J. Gao, “[Large language models: A survey](https://md.research.elamur.ai/free/en/papers/minaee-2024-large-language-models-survey/),” arXiv, 2 2024. [Online]. Available: https://arxiv.org/abs/2402.06196 [↗](https://arxiv.org/abs/2402.06196)
10. N. Shinn, F. Cassano, E. Berman, A. Gopinath, K. Narasimhan, and S. Yao, “[Reflexion: Language agents with verbal reinforcement learning](https://md.research.elamur.ai/free/en/papers/shinn-2023-reflexion-language-agents-verbal-reinforcement/),” arXiv, 3 2023. [Online]. Available: https://arxiv.org/abs/2303.11366 [↗](https://arxiv.org/abs/2303.11366)
11. L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Y. Wang, “Automatically correcting large language models: Surveying the landscape of diverse self-correction strategies,” arXiv, 8 2023. [Online]. Available: https://arxiv.org/abs/2308.03188 [↗](https://arxiv.org/abs/2308.03188)
12. A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, S. Wiegreffe, U. Alon, N. Dziri, S. Prabhumoye, Y. Yang, S. Gupta, B. P. Majumder, K. Hermann, S. Welleck, A. Yazdanbakhsh, and P. Clark, “Self-refine: Iterative refinement with self-feedback,” arXiv, 3 2023. [Online]. Available: [https://arxiv.org…](https://arxiv.org…) [↗](https://arxiv.org…)
13. J. Toy, J. MacAdam, and P. Tabor, “Metacognition is all you need? using introspection in generative agents to improve goal-directed behavior,” arXiv, 1 2024. [Online]. Available: https://arxiv.org/abs/2401.10910 — [arXiv:2401.10910](https://arxiv.org/abs/2401.10910) [↗](https://arxiv.org/abs/2401.10910)
14. Y. Wang and Y. Zhao, “Metacognitive prompting improves understanding in large language models,” arXiv, 8 2023. [Online]. Available: https://arxiv.org/abs/2308.05342 [↗](https://arxiv.org/abs/2308.05342)
15. A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “[Self-rag: Learning to retrieve, generate, and critique through self-reflection](https://md.research.elamur.ai/free/en/papers/asai-2023-self-rag-learning-retrieve-generate/),” arXiv, 10 2023. [Online]. Available: https://arxiv.org/abs/2310.11511v1 [↗](https://arxiv.org/abs/2310.11511v1)
16. G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, A. Anandkumar, U. Austin, and U. Madison, “Voyager: An open-ended embodied agent with large language models,” arXiv, 5 2023. [Online]. Available: https://arxiv.org/abs/2305.16291v2 [↗](https://arxiv.org/abs/2305.16291v2)
17. [Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, R. Zheng, X. Fan, X. Wang, L. Xiong, Y. Zhou, W. Wang, C. Jiang, Y. Zou, X. Liu, Z. Yin, S. Dou, R. Weng, W. Cheng, Q. Zhang, W. Qin, Y. Zheng, X. Qiu, X. Huang, and T. Gui, “The rise and potential of large language…](https://md.research.elamur.ai/free/en/papers/xi-2023-rise-potential-large-language/)
18. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “[React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/),” arXiv, 10 2022. [Online]. Available: https://arxiv.org/abs/2210.03629 [↗](https://arxiv.org/abs/2210.03629)
19. N. Miao, Y. W. Teh, and T. Rainforth, “Selfcheck: Using llms to zero-shot check their own step-by-step reasoning,” arXiv, 8 2023. [Online]. Available: https://arxiv.org/abs/2308.00436v3 [↗](https://arxiv.org/abs/2308.00436v3)
20. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. S. Openai, “[Webgpt: Browser-assisted question-answering with human feedback](https://md.research.elamur.ai/free/en/papers/nakano-2021-webgpt-browser-assisted-question-answering/),” arXiv, 12 2021. [Online]. Availabl…
21. T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “[Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/),” arXiv, 2 2023. [Online]. Available: https://arxiv.org/abs/2302.04761v1 [↗](https://arxiv.org/abs/2302.04761v1)
22. P. Lewis and et al., “Retrieval-augmented generation for knowledge-intensive nlp tasks,” arXiv, 5 2020. [Online]. Available: https://arxiv.org/abs/2005.11401 [↗](https://arxiv.org/abs/2005.11401)
23. Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, M. Wang, and H. Wang, “[Retrieval-augmented generation for large language models: A survey](https://md.research.elamur.ai/free/en/papers/gao-2023-retrieval-augmented-generation-large-language/),” arXiv, 12 2023. [Online]. Available: https://arxiv.org/abs/2312.10997v5 [↗](https://arxiv.org/abs/2312.10997v5)
24. W. Zhong, L. Guo, Q. Gao, H. Ye, and Y. Wang, “Memorybank: Enhancing large language models with long-term memory,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 19 724–19 731, 5 2023. [Online]. Available: https://arxiv.org/abs/2305.10250v3 [↗](https://arxiv.org/abs/2305.10250v3)
25. Z. Wang, A. Liu, H. Lin, J. Li, X. Ma, and Y. Liang, “Rat: Retrieval augmented thoughts elicit context-aware reasoning in long-horizon generation,” arXiv, 3 2024. [Online]. Available: https://arxiv.org/abs/2403.05313 [↗](https://arxiv.org/abs/2403.05313)
26. G. Tyen, H. Mansoor, V. Cărbune, P. Chen, and T. Mak, “Llms cannot find reasoning errors, but can correct them!” arXiv, 11 2023. [Online]. Available: https://arxiv.org/abs/2311.08516v2 — [arXiv:2311.08516](https://arxiv.org/abs/2311.08516) [↗](https://arxiv.org/abs/2311.08516v2)
27. P. Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” ArXiv, 3 2018. [Online]. Available: https://arxiv.org/abs/1803.05457 [↗](https://arxiv.org/abs/1803.05457)
28. R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hellaswag: Can a machine really finish your sentence?” in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019. [Online]. Available: https://arxiv.org/abs/1905.07830 [↗](https://arxiv.org/abs/1905.07830)
29. [A. Pal, L. K. Umapathi, and M. Sankarasubbu, “Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering,” in Proceedings of the Conference on Health, Inference, and Learning. PMLR, 2022, pp. 248–260. Online. Available:](https://md.research.elamur.ai/free/en/papers/pal-2022-medmcqa-large-scale-multi-subject/) [https://proceedings.mlr.press/v174/pal22a…](https://proceedings.mlr.press/v174/pal22a…) [↗](https://proceedings.mlr.press/v174/pal22a…)
30. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan, “[Agieval: A human-centric benchmark for evaluating foundation models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/),” ArXiv, 4 2023. [Online]. Available: https://arxiv.org/abs/2304.06364 [↗](https://arxiv.org/abs/2304.06364)
31. J. Liu, L. Cui, H. Liu, D. Huang, Y. Wang, and Y. Zhang, “[Logiqa: A challenge dataset for machine reading comprehension with logical reasoning](https://md.research.elamur.ai/free/en/papers/liu-2020-logiqa-challenge-dataset-machine-reading/),” in International Joint Conference on Artificial Intelligence, 2020. [Online]. Available: https://arxiv.org/abs/2007.08124 [↗](https://arxiv.org/abs/2007.08124)
32. S. Wang, Z. Liu, W. Zhong, M. Zhou, Z. Wei, Z. Chen, and N. Duan, “From lsat: The progress and challenges of complex reasoning,” IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 30, pp. 2201–2216, 8 2021. [Online]. Available: https://doi.org/10.1109/TASLP.2022.3164218 — [doi:10.1109/taslp.2022.3164218](https://doi.org/10.1109/taslp.2022.3164218) [↗](https://doi.org/10.1109/TASLP.2022.3164218)
33. Anthropic, “Introducing the next generation of claude anthropic,” 2024. [Online]. Available: [https://www.anthropic.com/news/claude-3-family](https://www.anthropic.com/news/claude-3-family) [↗](https://www.anthropic.com/news/claude-3-family)
34. ——, “The claude 3 model family: Opus, sonnet, haiku,” 2024. [Online]. Available: [https://www.anthropic.com/claude-3-model-card](https://www.anthropic.com/claude-3-model-card) [↗](https://www.anthropic.com/claude-3-model-card)
35. Cohere, “Command r+,” 2024. [Online]. Available: [https://docs.cohere.com/docs/command-r-plus](https://docs.cohere.com/docs/command-r-plus) [↗](https://docs.cohere.com/docs/command-r-plus)
36. ——, “Model card for c4ai command r+,” 2024. [Online]. Available: [https://huggingface.co/CohereForAI/](https://huggingface.co/CohereForAI/) c4ai-command-r-plus [↗](https://huggingface.co/CohereForAI/)
37. S. Pichai and D. Hassabis, “Introducing gemini: Google’s most capable ai model yet,” 2023. [Online]. Available: [https://blog.google/technology/ai/google-gemini-ai/](https://blog.google/technology/ai/google-gemini-ai/) [↗](https://blog.google/technology/ai/google-gemini-ai/)
38. Gemini-Team, “[Gemini: A family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/),” arXiv, 12 2023.
39. S. Pichai and D. Hassabis, “Introducing gemini 1.5, google’s next-generation ai model,” 2024. [Online]. Available: [https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/) [↗](https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/)
40. Gemini-Team, “[Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/),” 2024. [Online]. Available: https://arxiv.org/abs/2403.05530 [↗](https://arxiv.org/abs/2403.05530)
41. OpenAI, “Introducing chatgpt,” 11 2022. [Online]. Available: [https://openai.com/blog/chatgpt](https://openai.com/blog/chatgpt) [↗](https://openai.com/blog/chatgpt)
42. ——, “Models - openai api.” [Online]. Available: [https://platform.openai.com/docs/models/gpt-3-5-turbo](https://platform.openai.com/docs/models/gpt-3-5-turbo) [↗](https://platform.openai.com/docs/models/gpt-3-5-turbo)
43. ——, “Gpt-4,” 3 2023. [Online]. Available: [https://openai.com/research/gpt-4](https://openai.com/research/gpt-4) [↗](https://openai.com/research/gpt-4)
44. ——, “[Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/),” arXiv, 3 2023. [Online]. Available: https://arxiv.org/abs/2303.08774 [↗](https://arxiv.org/abs/2303.08774)
45. Meta, “Meta and microsoft introduce the next generation of llama \| meta,” 2023. [Online]. Available: [https://about.meta.com/news/2023/07/llama-2/](https://about.meta.com/news/2023/07/llama-2/) [↗](https://about.meta.com/news/2023/07/llama-2/)
46. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan,… [find](https://scholar.google.com/scholar?q=H%2E+Touvron%2C+L%2E+Martin%2C+K%2E+Stone%2C+P%2E+Albert%2C+A%2E+Almahairi%2C+Y%2E+Babaei%2C+N%2E+Bashlykov%2C+S%2E+Batra%2C+P%2E+Bhargava%2C+S%2E+Bhosale%2C+D%2E+Bikel%2C+L%2E+Blecher%2C+C%2E+C%2E+Ferrer%2C+M%2E+Chen%2C+G%2E+Cucurull%2C+D%2E+Esiobu%2C+J%2E+Fernandes%2C)
47. Mistral-AI-Team, “Au large \| mistral ai \| frontier ai in your hands,” 2024. [Online]. Available: [https://mistral.ai/news/mistral-large/](https://mistral.ai/news/mistral-large/) [↗](https://mistral.ai/news/mistral-large/)
48. S. M. Bsharat, A. Myrzakhan, and Z. Shen, “Principled instructions are all you need for questioning llama-1/2, gpt-3.5/4,” arXiv, 12 2023. [Online]. Available: https://arxiv.org/abs/2312.16171 — [arXiv:2312.16171](https://arxiv.org/abs/2312.16171) [↗](https://arxiv.org/abs/2312.16171)
49. M. Renze and E. Guven, “The benefits of a concise chain of thought on problem-solving in large language models,” arXiv, 1 2024. [Online]. Available: https://arxiv.org/abs/2401.05618v1 — [arXiv:2401.05618](https://arxiv.org/abs/2401.05618) [↗](https://arxiv.org/abs/2401.05618v1)
50. ——, “[The effect of sampling temperature on problem solving in large language models](https://md.research.elamur.ai/free/en/papers/renze-2024-effect-sampling-temperature-problem-solving/),” arXiv, 2 2024. [Online]. Available: https://arxiv.org/abs/2402.05201v1 [↗](https://arxiv.org/abs/2402.05201v1)
51. Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, pp. 153–157, 6 1947. [Online]. Available: https://doi.org/10.1007/BF02295996 — [doi:10.1007/bf02295996](https://doi.org/10.1007/bf02295996) [↗](https://doi.org/10.1007/BF02295996)

*Updated 24 October 2026*