---
title: "Imitate, Explore, and Self-Improve — paper by Min et al. (2024)"
description: "This report replaces that design with STILL-2, which generates thought and solution in one decoding… 34 references, cited by 64 papers here."
authors: ["Yingqian Min", "Zhipeng Chen", "Jinhao Jiang", "Jie Chen", "Jia Deng", "Yiwen Hu", "Yiru Tang", "Jiapeng Wang", "Xiaoxue Cheng", "Huatong Song", "Wayne Xin Zhao", "Zheng Liu", "Zhongyuan Wang", "Ji-Rong Wen"]
year: 2024
venue: "arXiv:2412.09413"
source: "https://arxiv.org/abs/2412.09413"
doi: "10.48550/arxiv.2412.09413"
arxiv: "2412.09413"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/min-2024-imitate-explore-self-improve-reproduction/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/min-2024-imitate-explore-self-improve-reproduction/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems

# Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems

**Authors:** [Yingqian Min](https://md.research.elamur.ai/free/en/authors/yingqian-min/), [Zhipeng Chen](https://md.research.elamur.ai/free/en/authors/zhipeng-chen/), [Jinhao Jiang](https://md.research.elamur.ai/free/en/authors/jinhao-jiang/), [Jie Chen](https://md.research.elamur.ai/free/en/authors/jie-chen/), [Jia Deng](https://md.research.elamur.ai/free/en/authors/jia-hao-deng/), Yiwen Hu, Yiru Tang, [Jiapeng Wang](https://md.research.elamur.ai/free/en/authors/jiapeng-wang/), [Xiaoxue Cheng](https://md.research.elamur.ai/free/en/authors/xiaoxue-cheng/), [Huatong Song](https://md.research.elamur.ai/free/en/authors/huatong-song/), [Wayne Xin Zhao](https://md.research.elamur.ai/free/en/authors/wayne-xin-zhao/), [Zheng Liu](https://md.research.elamur.ai/free/en/authors/zheng-liu/), Zhongyuan Wang and [Ji-Rong Wen](https://md.research.elamur.ai/free/en/authors/ji-rong-wen/) · **Year:** 2024 · **Venue:** arXiv:2412.09413 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2412.09413) · [PDF](https://arxiv.org/pdf/2412.09413) · [DOI 10.48550/arxiv.2412.09413](https://doi.org/10.48550/arxiv.2412.09413) · [arXiv 2412.09413](https://arxiv.org/abs/2412.09413) · [OpenAlex](https://openalex.org/W4405355299)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [64](https://md.research.elamur.ai/free/en/papers/min-2024-imitate-explore-self-improve-reproduction/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Industrial "slow-thinking" models such as OpenAI's o1 produce long internal reasoning before answering, but their training recipes are not public. Open reproductions often stay within one domain or rely on weak base models. The authors' own earlier attempt combined a policy model, a reward model and tree search. It suffered from poor cross-domain transfer of the reward model and slow inference. This report replaces that design with STILL-2, which generates thought and solution in one decoding pass. Training has three stages. First, the model imitates a small set of long-form reasoning traces distilled from the public o1-like systems of DeepSeek and Qwen. Second, it explores hard problems by sampling many rollouts and keeping those whose final answer matches the ground truth. Third, it retrains on these trajectories with SFT or DPO. On MATH-OAI, AIME and GPQA, distillation with about 3,900 examples comes close to some industry systems. The explore-and-improve loop also performs well starting from only 1,100 seed examples. The work shows that long reasoning can be unlocked cheaply, without reward models or search at inference time.

## Key points
- Code and data are published in the RUCAIBox/Slow_Thinking_with_LLMs GitHub repository. STILL-2 is the second model of the "Slow Thinking with LLMs" project.
- Evaluation uses MATH-OAI, AIME and GPQA. Distilled training is scaled to 3,900 demonstrations, while the self-improvement variant starts from 1,100 seed demonstrations.
- In the data-mixture analysis, removing hard math problems (AIME/Olympiad level) clearly lowers scores, most of all on AIME. These problems also come with longer thought traces.
- Training on math data alone still gives strong results on all three benchmarks, including the science-focused GPQA. This suggests the slow-thinking ability transfers across domains.
- The DPO variant was modified to align only the thought part and to add an SFT loss. In the ablation, the strongest run aligned only the thought, with no SFT term (average 61.0, AIME 46.7). Results varied between runs, and gains from offline self-improvement came mostly in the early iterations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Renmin University of China](https://md.research.elamur.ai/free/en/universities/renmin-university-of-china/) — University, Beijing, China · 139 works · 17,752 citations
- [Beijing Academy of Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/beijing-academy-of-artificial-intelligence/) — Research institute, Beijing, China · 46 works · 8,682 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang and Ji-Rong Wen. 2024. Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems. arXiv:2412.09413.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{min2024imitate,
  title = {{Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems}},
  author = {Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang and Ji-Rong Wen},
  year = {2024},
  doi = {10.48550/arxiv.2412.09413},
  eprint = {2412.09413},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2412.09413}
}
```

- [← previous C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness](https://md.research.elamur.ai/free/en/papers/kang-2024-c3ot-generating-shorter-chain-thought/)
- [next → JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks](https://md.research.elamur.ai/free/en/papers/luo-2024-jailbreakv-benchmark-assessing-robustness-multimodal/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

34 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large l… — [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)
2. Kahneman Daniel. Thinking, fast and slow. 2017. [find](https://scholar.google.com/scholar?q=Kahneman+Daniel%2E+Thinking%2C+fast+and+slow%2E+2017%2E)
3. Siwei Wu, Zhongyuan Peng, Xinrun Du, Tuney Zheng, Minghao Liu, Jialong Wu, Jiachen Ma, Yizhi Li, Jian Yang, Wangchunshu Zhou, Qunshu Lin, Junbo Zhao, Zhaoxiang Zhang, Wenhao Huang, Ge Zhang, Chenghua Lin, and Jiaheng Liu. A comparative study on reasoning patterns of openai’s o1 model. CoRR, abs/2410… — [arXiv:2410.13639](https://arxiv.org/abs/2410.13639) [find](https://scholar.google.com/scholar?q=Siwei+Wu%2C+Zhongyuan+Peng%2C+Xinrun+Du%2C+Tuney+Zheng%2C+Minghao+Liu%2C+Jialong+Wu%2C+Jiachen+Ma%2C+Yizhi+Li%2C+Jian+Yang%2C+Wangchunshu+Zhou%2C+Qunshu+Lin%2C+Junbo+Zhao%2C+Zhaoxiang+Zhang%2C+Wenhao+Huang%2C+Ge+Zhang%2C+Chenghua+)
4. Tianyang Zhong, Zhengliang Liu, Yi Pan, Yutong Zhang, Yifan Zhou, Shizhe Liang, Zihao Wu, Yanjun Lyu, Peng Shu, Xiaowei Yu, Chao Cao, Hanqi Jiang, Hanxu Chen, Yiwei Li, Junhao Chen, Huawen Hu, Yihen Liu, Huaqin Zhao, Shaochen Xu, Haixing Dai, Lin Zhao, Ruidong Zhang, Wei Zhao, Zhenyuan Yang, Jingyua… — [Evaluation of OpenAI o1: Opportunities and Challenges of AGI](https://md.research.elamur.ai/free/en/papers/zhong-2024-evaluation-openai-o1-opportunities-challenges/)
5. OpenAI. Learning to reason with llms, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Learning+to+reason+with+llms%2C+2024%2E)
6. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In NeurIPS, 2022.
7. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. [Qwen2.5-math technical report: Toward mathematical expert model via self-improvement](https://md.research.elamur.ai/free/en/papers/yang-2024-qwen2-5-math-technical-report/). CoRR, a…
8. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. [Deepseekmath: Pushing the limits of mathematical reasoning in open language models](https://md.research.elamur.ai/free/en/papers/shao-2024-deepseekmath-pushing-limits-mathematical-reasoning/). CoRR, abs/2402.03300, 2024.
9. Jinhao Jiang, Zhipeng Chen, Yingqian Min, Jie Chen, Xiaoxue Cheng, Jiapeng Wang, Yiru Tang, Haoxiang Sun, Jia Deng, Wayne Xin Zhao, et al. Technical report: Enhancing llm reasoning with reward-guided tree search. CoRR, abs/2411.11694, 2024. — [arXiv:2411.11694](https://arxiv.org/abs/2411.11694) [find](https://scholar.google.com/scholar?q=Jinhao+Jiang%2C+Zhipeng+Chen%2C+Yingqian+Min%2C+Jie+Chen%2C+Xiaoxue+Cheng%2C+Jiapeng+Wang%2C+Yiru+Tang%2C+Haoxiang+Sun%2C+Jia+Deng%2C+Wayne+Xin+Zhao%2C+et+al%2E+Technical+report%3A+Enhancing+llm+reasoning+with+reward%2Dguided+)
10. Di Zhang, Jianbo Wu, Jingdi Lei, Tong Che, Jiatong Li, Tong Xie, Xiaoshui Huang, Shufei Zhang, Marco Pavone, Yuqiang Li, Wanli Ouyang, and Dongzhan Zhou. Llama-berry: Pairwise optimization for o1-like olympiad-level mathematical reasoning. CoRR, abs/2410.02884, 2024. [find](https://scholar.google.com/scholar?q=Di+Zhang%2C+Jianbo+Wu%2C+Jingdi+Lei%2C+Tong+Che%2C+Jiatong+Li%2C+Tong+Xie%2C+Xiaoshui+Huang%2C+Shufei+Zhang%2C+Marco+Pavone%2C+Yuqiang+Li%2C+Wanli+Ouyang%2C+and+Dongzhan+Zhou%2E+Llama%2Dberry%3A+Pairwise+optimization+for+o1%2Dlike)
11. Yuxiang Zhang, Shangxi Wu, Yuqi Yang, Jiangming Shu, Jinlin Xiao, Chao Kong, and Jitao Sang. o1-coder: an o1 replication for coding. CoRR, abs/2412.00154, 2024. — [arXiv:2412.00154](https://arxiv.org/abs/2412.00154) [find](https://scholar.google.com/scholar?q=Yuxiang+Zhang%2C+Shangxi+Wu%2C+Yuqi+Yang%2C+Jiangming+Shu%2C+Jinlin+Xiao%2C+Chao+Kong%2C+and+Jitao+Sang%2E+o1%2Dcoder%3A+an+o1+replication+for+coding%2E+CoRR%2C+abs%2F2412%2E00154%2C+2024%2E+%E2%80%94+%5BarXiv%3A2412%2E00154%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
12. Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, and Pengfei Liu. O1 replication journey: A strategic progress report – part 1. CoRR, 2024. — [O1 Replication Journey: A Strategic Progress Report -- Part 1](https://md.research.elamur.ai/free/en/papers/qin-2024-o1-replication-journey-strategic-progress/)
13. Yu Zhao, Huifeng Yin, Bo Zeng, Hao Wang, Tianqi Shi, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. [Marco-o1: Towards open reasoning models for open-ended solutions](https://md.research.elamur.ai/free/en/papers/zhao-2024-marco-o1-towards-open-reasoning/). CoRR, abs/2411.14405, 2024.
14. Skywork o1 Team. Skywork-o1 open series. [https://huggingface.co/Skywork](https://huggingface.co/Skywork), November 2024. [↗](https://huggingface.co/Skywork)
15. DeepSeek Team. Deepseek-r1-lite-preview is now live: unleashing supercharged reasoning power!, November 2024. [find](https://scholar.google.com/scholar?q=DeepSeek+Team%2E+Deepseek%2Dr1%2Dlite%2Dpreview+is+now+live%3A+unleashing+supercharged+reasoning+power%21%2C+November+2024%2E)
16. Qwen Team. Qwq: Reflect deeply on the boundaries of the unknown, November 2024. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+Qwq%3A+Reflect+deeply+on+the+boundaries+of+the+unknown%2C+November+2024%2E)
17. Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, and Pengfei Liu. O1 replication journey–part 2: Surpassing o1-preview through simple distillation, big progress or bitter lesson? arXiv preprint arXiv:2411.16489, 2024. [find](https://scholar.google.com/scholar?q=Zhen+Huang%2C+Haoyang+Zou%2C+Xuefeng+Li%2C+Yixiu+Liu%2C+Yuxiang+Zheng%2C+Ethan+Chern%2C+Shijie+Xia%2C+Yiwei+Qin%2C+Weizhe+Yuan%2C+and+Pengfei+Liu%2E+O1+replication+journey%E2%80%93part+2%3A+Surpassing+o1%2Dpreview+through+simple+dis)
18. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In ICLR. OpenReview.net, 2024. — [Let's Verify Step by Step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/)
19. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). CoRR, abs/2311.12022, 2023.
20. Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. [Quiet-star: Language models can teach themselves to think before speaking](https://md.research.elamur.ai/free/en/papers/al-2024-quiet-star-language-models-teach/). CoRR, abs/2403.09629, 2024.
21. David Herel and Tomás Mikolov. Thinking tokens for language modeling. CoRR, abs/2405.08644, 2024. — [arXiv:2405.08644](https://arxiv.org/abs/2405.08644) [find](https://scholar.google.com/scholar?q=David+Herel+and+Tom%C3%A1s+Mikolov%2E+Thinking+tokens+for+language+modeling%2E+CoRR%2C+abs%2F2405%2E08644%2C+2024%2E+%E2%80%94+%5BarXiv%3A2405%2E08644%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2405%2E08644%29)
22. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In ICLR. OpenReview.net, 2023.
23. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NeurIPS, 2023. [find](https://scholar.google.com/scholar?q=Shunyu+Yao%2C+Dian+Yu%2C+Jeffrey+Zhao%2C+Izhak+Shafran%2C+Tom+Griffiths%2C+Yuan+Cao%2C+and+Karthik+Narasimhan%2E+Tree+of+thoughts%3A+Deliberate+problem+solving+with+large+language+models%2E+In+NeurIPS%2C+2023%2E)
24. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 2024. [find](https://scholar.google.com/scholar?q=Jia+Li%2C+Edward+Beeching%2C+Lewis+Tunstall%2C+Ben+Lipkin%2C+Roman+Soletskyi%2C+Shengyi+Huang%2C+Kashif+Rasul%2C+Longhui+Yu%2C+Albert+Q+Jiang%2C+Ziju+Shen%2C+et+al%2E+Numinamath%3A+The+largest+public+dataset+in+ai4maths+with)
25. Zachary Ankner, Cody Blakeney, Kartik Sreenivasan, Max Marion, Matthew L. Leavitt, and Mansheej Paul. Perplexed by perplexity: Perplexity-based data pruning with small reference models. CoRR, abs/2405.20541, 2024. — [arXiv:2405.20541](https://arxiv.org/abs/2405.20541) [find](https://scholar.google.com/scholar?q=Zachary+Ankner%2C+Cody+Blakeney%2C+Kartik+Sreenivasan%2C+Max+Marion%2C+Matthew+L%2E+Leavitt%2C+and+Mansheej+Paul%2E+Perplexed+by+perplexity%3A+Perplexity%2Dbased+data+pruning+with+small+reference+models%2E+CoRR%2C+abs%2F2405)
26. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Chuanqi Tan, and Chang Zhou. [Scaling relationship on learning mathematical reasoning with large language models](https://md.research.elamur.ai/free/en/papers/yuan-2023-scaling-relationship-learning-mathematical-reasoning/). CoRR, abs/2308.01825, 2023.
27. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. [Star: Bootstrapping reasoning with reasoning](https://md.research.elamur.ai/free/en/papers/zelikman-2022-star-bootstrapping-reasoning-reasoning/). In NeurIPS, 2022.
28. Qwen Team. Qwen2.5: A party of foundation models, September 2024. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+Qwen2%2E5%3A+A+party+of+foundation+models%2C+September+2024%2E)
29. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. [Direct preference optimization: Your language model is secretly a reward model](https://md.research.elamur.ai/free/en/papers/rafailov-2023-direct-preference-optimization-language-model/). In NeurIPS, 2023.
30. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. [find](https://scholar.google.com/scholar?q=John+Schulman%2C+Filip+Wolski%2C+Prafulla+Dhariwal%2C+Alec+Radford%2C+and+Oleg+Klimov%2E+Proximal+policy+optimization+algorithms%2E+CoRR%2C+abs%2F1707%2E06347%2C+2017%2E)
31. Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. Online iterative reinforcement learning from human feedback with general preference model. CoRR, abs/2402.07314, 2024. — [arXiv:2402.07314](https://arxiv.org/abs/2402.07314) [find](https://scholar.google.com/scholar?q=Chenlu+Ye%2C+Wei+Xiong%2C+Yuheng+Zhang%2C+Nan+Jiang%2C+and+Tong+Zhang%2E+Online+iterative+reinforcement+learning+from+human+feedback+with+general+preference+model%2E+CoRR%2C+abs%2F2402%2E07314%2C+2024%2E+%E2%80%94+%5BarXiv%3A2402%2E0731)
32. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Joaquin Vanschoren and Sai-Kit Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datas…
33. OpenAI. Hello gpt-4o, 2024. [find](https://scholar.google.com/scholar?q=OpenAI%2E+Hello+gpt%2D4o%2C+2024%2E)
34. Anthropic. Claude 3.5 sonnet, 2024. Problem: Convert the point (0, 3) in rectangular coordinates to polar coordinates. Enter your answer in the form (r, θ), where r \> 0 and 0 ≤ θ \\< 2π. Output: \\<\|begin_of_thought\|\> [find](https://scholar.google.com/scholar?q=Anthropic%2E+Claude+3%2E5+sonnet%2C+2024%2E+Problem%3A+Convert+the+point+%280%2C+3%29+in+rectangular+coordinates+to+polar+coordinates%2E+Enter+your+answer+in+the+form+%28r%2C+%CE%B8%29%2C+where+r+%5C%3E+0+and+0+%E2%89%A4+%CE%B8+%5C%3C+2%CF%80%2E+Output%3A+%5C%3C%5C%7Cb)

*Updated 2 November 2026*