---
title: "ARISE — paper by Yin et al. (2026) — summary and ideas"
description: "Reasoning models can spend more inference compute (\"test-time scaling\"), but there has been no… 63 references, cited by 1 paper here."
authors: ["Zhangyue Yin", "Qiushi Sun", "Zhiyuan Zeng", "Zhiyuan Yu", "Qipeng Guo", "Xuanjing Huang", "Xipeng Qiu"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.289/"
doi: "10.18653/v1/2026.findings-acl.289"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2026-arise-adaptive-resolution-aware-metric/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2026-arise-adaptive-resolution-aware-metric/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/) / ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models

# ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models

**Authors:** [Zhangyue Yin](https://md.research.elamur.ai/free/en/authors/zhangyue-yin/), [Qiushi Sun](https://md.research.elamur.ai/free/en/authors/qiushi-sun/), [Zhiyuan Zeng](https://md.research.elamur.ai/free/en/authors/zhiyuan-zeng/), [Zhiyuan Yu](https://md.research.elamur.ai/free/en/authors/zhiyuan-yu/), [Qipeng Guo](https://md.research.elamur.ai/free/en/authors/qipeng-guo/), [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) and [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.289/) · [PDF](https://aclanthology.org/2026.findings-acl.289.pdf) · [DOI 10.18653/v1/2026.findings-acl.289](https://doi.org/10.18653/v1/2026.findings-acl.289) · [OpenAlex](https://openalex.org/W4414981003)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/yin-2026-arise-adaptive-resolution-aware-metric/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Reasoning models can spend more inference compute ("test-time scaling"), but there has been no dependable way to compare how well different models turn extra tokens into better answers. The earlier slope-based Scaling metric (Muennighoff et al., 2025) works only on aggregate accuracy. It ignores individual problems that flip from right to wrong, and under negative scaling it can give a higher score to a model that spends more tokens for worse results.

ARISE fixes this by following each sample across successive scaling iterations. When a sample becomes correct, it earns a reward equal to the ratio of previous to current tokens. When a correct sample becomes wrong, it receives a larger penalty equal to the inverse ratio. Because the weights are ratios, the score adjusts to problem difficulty. A dynamic sampling scheme adds repeated runs for samples whose accuracy or token counts fluctuate.

The authors tested ARISE on math, science, code, agentic and multimodal benchmarks. It proved both more sensitive to real degradation and more robust to noise than the slope metric, and Claude Opus showed the strongest scaling. This makes ARISE a practical tool for ranking reasoning models by how efficiently they scale.

## Key points
- **Scoring rule:** a sample's ARISE score sums its accuracy changes, each weighted by a token ratio. Correct-to-wrong flips get a penalty above 1 in magnitude, so wasting compute is punished asymmetrically. The final score averages over all samples.
- **Dynamic sampling:** the number of repeated runs per sample adapts to observed variance, with a cap of m_max = 10. Beyond that cap, extra runs bring little benefit and can roughly double the cost.
- **Perturbation tests on AIME:**
  - Sensitivity: flipping 10% of correct answers to incorrect lowered ARISE by 27.1%, versus 3.5% for the Scaling metric, a 7.7× higher responsiveness.
  - Robustness: under Gaussian token noise up to 0.3µ, ARISE's coefficient of variation stayed below 0.13, versus 0.31 for the Scaling metric, 2.4× more stable.
- **Model ranking:** among contemporary reasoning models, Claude Opus scored highest on ARISE and remained stable across all task domains.
- **Limitations:**
  - Agentic evaluation covers only the mock, airline, retail and telecom scenarios of τ²-bench.
  - All datasets are English-only, and the authors call for cross-lingual evaluation of test-time scaling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations
- [University of Hong Kong](https://md.research.elamur.ai/free/en/universities/university-of-hong-kong/) — University, Pok Fu Lam, China · 118 works · 13,496 citations
- [Nanjing University](https://md.research.elamur.ai/free/en/universities/nanjing-university/) — University, Nanjing, China · 99 works · 8,512 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Zhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Zhiyuan Yu, Qipeng Guo, Xuanjing Huang and Xipeng Qiu. 2026. ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yin2026arise,
  title = {{ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models}},
  author = {Zhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Zhiyuan Yu, Qipeng Guo, Xuanjing Huang and Xipeng Qiu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.289},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.289}
}
```

- [← previous ARGSBASE: A Multi-Agent Interface for Structured Human–AI Deliberation](https://md.research.elamur.ai/free/en/papers/turkstra-2026-argsbase-multi-agent-interface-structured/)
- [next → ART: Adaptive Reasoning Trees for Explainable Claim Verification](https://md.research.elamur.ai/free/en/papers/wadhwa-2026-art-adaptive-reasoning-trees-explainable/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/10/)

## References

63 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. [gpt-oss-120b & gpt-oss-20b model card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/). arXiv preprint arXiv:2508.10925.
2. Pranjal Aggarwal and Sean Welleck. 2025. [L1: Controlling how long a reasoning model thinks with reinforcement learning](https://md.research.elamur.ai/free/en/papers/aggarwal-2025-l1-controlling-long-reasoning-model/). arXiv preprint arXiv:2503.04697.
3. Anthropic. 2025a. Claude opus 4.1. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025a%2E+Claude+opus+4%2E1%2E)
4. Anthropic. 2025b. Introducing claude 4. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025b%2E+Introducing+claude+4%2E)
5. Vidhisha Balachandran, Jingya Chen, Lingjiao Chen, Shivam Garg, Neel Joshi, Yash Lara, John Langford, Besmira Nushi, Vibhav Vineet, Yue Wu, et al. 2025. Inference-time scaling for complex tasks: Where we stand and what lies ahead. arXiv preprint arXiv:2504.00294. [find](https://scholar.google.com/scholar?q=Vidhisha+Balachandran%2C+Jingya+Chen%2C+Lingjiao+Chen%2C+Shivam+Garg%2C+Neel+Joshi%2C+Yash+Lara%2C+John+Langford%2C+Besmira+Nushi%2C+Vibhav+Vineet%2C+Yue+Wu%2C+et+al%2E+2025%2E+Inference%2Dtime+scaling+for+complex+tasks%3A+Where)
6. Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. 2025. [Matharena: Evaluating llms on uncontaminated math competitions](https://md.research.elamur.ai/free/en/papers/balunovic-2025-matharena-evaluating-llms-uncontaminated-math/).
7. Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 2025. [τ 2 -bench: Evaluating conversational agents in a dual-control environment](https://md.research.elamur.ai/free/en/papers/barres-2025-tau-2-bench-evaluating-conversational/). Preprint, arXiv:2506.07982.
8. Aman Bhargava, Cameron Witkowski, Shi-Zhuo Looi, and Matt Thomson. 2023. What’s the magic word? a control theory of llm prompting. arXiv preprint arXiv:2310.04444. — [arXiv:2310.04444](https://arxiv.org/abs/2310.04444) [find](https://scholar.google.com/scholar?q=Aman+Bhargava%2C+Cameron+Witkowski%2C+Shi%2DZhuo+Looi%2C+and+Matt+Thomson%2E+2023%2E+What%E2%80%99s+the+magic+word%3F+a+control+theory+of+llm+prompting%2E+arXiv+preprint+arXiv%3A2310%2E04444%2E+%E2%80%94+%5BarXiv%3A2310%2E04444%5D%28https%3A%2F%2Farxiv%2Eo)
9. Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Ré, and Azalia Mirhoseini. 2024. [Large language monkeys: Scaling inference compute with repeated sampling](https://md.research.elamur.ai/free/en/papers/brown-2024-large-language-monkeys-scaling-inference/). arXiv preprint arXiv:2407.21787.
10. Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. 2025. [Open-reasoner-zero: An open source approach to scaling up reinforcement learning on the base model](https://md.research.elamur.ai/free/en/papers/hu-2025-open-reasoner-zero-open-source/). Preprint, arXiv:2503.24290.
11. Souradip Chakraborty, Mohammadreza Pourreza, Ruoxi Sun, Yiwen Song, Nino Scherrer, Furong Huang, Amrit Singh Bedi, Ahmad Beirami, Jindong Gu, Hamid Palangi, and Tomas Pfister. 2025. On the role of feedback in test-time scaling of agentic ai workflows. Preprint, arXiv:2504.01931. — [arXiv:2504.01931](https://arxiv.org/abs/2504.01931) [find](https://scholar.google.com/scholar?q=Souradip+Chakraborty%2C+Mohammadreza+Pourreza%2C+Ruoxi+Sun%2C+Yiwen+Song%2C+Nino+Scherrer%2C+Furong+Huang%2C+Amrit+Singh+Bedi%2C+Ahmad+Beirami%2C+Jindong+Gu%2C+Hamid+Palangi%2C+and+Tomas+Pfister%2E+2025%2E+On+the+role+of+fee)
12. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. 2024. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). arXiv preprint arXiv:2412.16720.
13. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
14. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). In The Thirteenth International Conference on Learning Representatio…
15. Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. 2024a. [Do not think that much for 2+ 3=?](https://md.research.elamur.ai/free/en/papers/chen-2024-think-that-much-2-3/) on the overthinking of o1-like llms. arXiv preprint arXiv:2412.21187.
16. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In ICLR.
17. Yanxi Chen, Xuchen Pan, Yaliang Li, Bolin Ding, and Jingren Zhou. 2024b. Simple and provable scaling laws for the test-time compute of large language models. arXiv preprint arXiv:2411.19477. — [arXiv:2411.19477](https://arxiv.org/abs/2411.19477) [find](https://scholar.google.com/scholar?q=Yanxi+Chen%2C+Xuchen+Pan%2C+Yaliang+Li%2C+Bolin+Ding%2C+and+Jingren+Zhou%2E+2024b%2E+Simple+and+provable+scaling+laws+for+the+test%2Dtime+compute+of+large+language+models%2E+arXiv+preprint+arXiv%3A2411%2E19477%2E+%E2%80%94+%5BarXiv%3A)
18. Cheng-Han Chiang and Hung-yi Lee. 2024. [Over-reasoning and redundant calculation of large language models](https://md.research.elamur.ai/free/en/papers/chiang-2024-over-reasoning-redundant-calculation-large/). In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–169, St. Julian’s, Malta. Association for Comp…
19. DeepSeek-AI. 2024. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+Preprint%2C+arXiv%3A2412%2E19437%2E)
20. DeepSeek-AI. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). Preprint, arXiv:2501.12948.
21. Shihan Dou, Ming Zhang, Chenhao Huang, Jiayi Chen, Feng Chen, Shichun Liu, Yan Liu, Chenxiao Liu, CHENG ZHONG, Zongzhang Zhang, et al. Evalearn: Quantifying the learning capability and efficiency of llms via sequential problem solving. In The Thirty-ninth Annual Conference on Neural Information Proc… [find](https://scholar.google.com/scholar?q=Shihan+Dou%2C+Ming+Zhang%2C+Chenhao+Huang%2C+Jiayi+Chen%2C+Feng+Chen%2C+Shichun+Liu%2C+Yan+Liu%2C+Chenxiao+Liu%2C+CHENG+ZHONG%2C+Zongzhang+Zhang%2C+et+al%2E+Evalearn%3A+Quantifying+the+learning+capability+and+efficiency+of+l)
22. Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, et al. 2026. Cl-bench: A benchmark for context learning. arXiv preprint arXiv:2602.03587. — [arXiv:2602.03587](https://arxiv.org/abs/2602.03587) [find](https://scholar.google.com/scholar?q=Shihan+Dou%2C+Ming+Zhang%2C+Zhangyue+Yin%2C+Chenhao+Huang%2C+Yujiong+Shen%2C+Junzhe+Wang%2C+Jiayi+Chen%2C+Yuchen+Ni%2C+Junjie+Ye%2C+Cheng+Zhang%2C+et+al%2E+2026%2E+Cl%2Dbench%3A+A+benchmark+for+context+learning%2E+arXiv+preprint+a)
23. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. [Measuring mathematical problem solving with the MATH dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
24. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. [Efficient memory management for large language model serving with pagedattention](https://md.research.elamur.ai/free/en/papers/kwon-2023-efficient-memory-management-large-language/). In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
25. Dacheng Li, Shiyi Cao, Chengkun Cao, Xiuyu Li, Shangyin Tan, Kurt Keutzer, Jiarong Xing, Joseph E Gonzalez, and Ion Stoica. 2025. S*: Test time scaling for code generation. arXiv preprint arXiv:2502.14382. [find](https://scholar.google.com/scholar?q=Dacheng+Li%2C+Shiyi+Cao%2C+Chengkun+Cao%2C+Xiuyu+Li%2C+Shangyin+Tan%2C+Kurt+Keutzer%2C+Jiarong+Xing%2C+Joseph+E+Gonzalez%2C+and+Ion+Stoica%2E+2025%2E+S%2A%3A+Test+time+scaling+for+code+generation%2E+arXiv+preprint+arXiv%3A2502%2E1)
26. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097.
27. Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2024. [Let’s verify step by step](https://md.research.elamur.ai/free/en/papers/lightman-2023-let-s-verify-step-step/). In The Twelfth International Conference on Learning Representations.
28. Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. 2024. [Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/). In The Twelfth International Conference on Learning Representations.
29. Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. 2025a. O1-pruner: Length-harmonizing fine-tuning for o1-like reasoning pruning. arXiv preprint arXiv:2501.12570. [find](https://scholar.google.com/scholar?q=Haotian+Luo%2C+Li+Shen%2C+Haiying+He%2C+Yibo+Wang%2C+Shiwei+Liu%2C+Wei+Li%2C+Naiqiang+Tan%2C+Xiaochun+Cao%2C+and+Dacheng+Tao%2E+2025a%2E+O1%2Dpruner%3A+Length%2Dharmonizing+fine%2Dtuning+for+o1%2Dlike+reasoning+pruning%2E+arXiv+prep)
30. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025b. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog. [find](https://scholar.google.com/scholar?q=Michael+Luo%2C+Sijun+Tan%2C+Justin+Wong%2C+Xiaoxiang+Shi%2C+William+Y%2E+Tang%2C+Manan+Roongta%2C+Colin+Cai%2C+Jeffrey+Luo%2C+Tianjun+Zhang%2C+Li+Erran+Li%2C+Raluca+Ada+Popa%2C+and+Ion+Stoica%2E+2025b%2E+Deepscaler%3A+Surpassing+o)
31. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/). Preprint, arXiv:2501.19393.
32. OpenAI. 2025a. Gpt-5 is here. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025a%2E+Gpt%2D5+is+here%2E)
33. OpenAI. 2025b. Introducing openai o3 and o4-mini. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025b%2E+Introducing+openai+o3+and+o4%2Dmini%2E)
34. OpenAI. 2025c. Introducing openai o3 and o4-mini. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025c%2E+Introducing+openai+o3+and+o4%2Dmini%2E)
35. OpenAI. 2025d. Openai o3-mini. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025d%2E+Openai+o3%2Dmini%2E)
36. Shishir G. Patil, Huanzhi Mao, Charlie Cheng-Jie Ji, Fanjia Yan, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. 2025. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Shishir+G%2E+Patil%2C+Huanzhi+Mao%2C+Charlie+Cheng%2DJie+Ji%2C+Fanjia+Yan%2C+Vishnu+Suresh%2C+Ion+Stoica%2C+and+Joseph+E%2E+Gonzalez%2E+2025%2E+The+berkeley+function+calling+leaderboard+%28bfcl%29%3A+From+tool+use+to+agentic+eva)
37. Nishat Raihan, Antonios Anastasopoulos, and Marcos Zampieri. 2024. mhumaneval–a multilingual benchmark to evaluate large language models for code generation. arXiv preprint arXiv:2410.15037. [find](https://scholar.google.com/scholar?q=Nishat+Raihan%2C+Antonios+Anastasopoulos%2C+and+Marcos+Zampieri%2E+2024%2E+mhumaneval%E2%80%93a+multilingual+benchmark+to+evaluate+large+language+models+for+code+generation%2E+arXiv+preprint+arXiv%3A2410%2E15037%2E)
38. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
39. Sheikh Shafayat, Eunsu Kim, Juhyun Oh, and Alice Oh. 2024. Multi-FAct: Assessing factuality of multilingual LLMs using FActscore. In First Conference on Language Modeling. [find](https://scholar.google.com/scholar?q=Sheikh+Shafayat%2C+Eunsu+Kim%2C+Juhyun+Oh%2C+and+Alice+Oh%2E+2024%2E+Multi%2DFAct%3A+Assessing+factuality+of+multilingual+LLMs+using+FActscore%2E+In+First+Conference+on+Language+Modeling%2E)
40. Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2025. [Scaling llm test-time compute optimally can be more effective than scaling model parameters](https://md.research.elamur.ai/free/en/papers/snell-2024-scaling-llm-test-time-compute/). In The Thirteenth International Conference on Learning Representations. OpenReview.
41. Jiankai Sun, Chuanyang Zheng, Enze Xie, Zhengying Liu, Ruihang Chu, Jianing Qiu, Jiaqi Xu, Mingyu Ding, Hongyang Li, Mengzhe Geng, et al. 2025. A survey of reasoning with foundation models: Concepts, methodologies, and outlook. ACM Computing Surveys, 57(11):1–43. [find](https://scholar.google.com/scholar?q=Jiankai+Sun%2C+Chuanyang+Zheng%2C+Enze+Xie%2C+Zhengying+Liu%2C+Ruihang+Chu%2C+Jianing+Qiu%2C+Jiaqi+Xu%2C+Mingyu+Ding%2C+Hongyang+Li%2C+Mengzhe+Geng%2C+et+al%2E+2025%2E+A+survey+of+reasoning+with+foundation+models%3A+Concepts%2C+)
42. Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, et al. 2024. A survey of neural code intelligence: Paradigms, advances and beyond. arXiv preprint arXiv:2403.14734. — [arXiv:2403.14734](https://arxiv.org/abs/2403.14734) [find](https://scholar.google.com/scholar?q=Qiushi+Sun%2C+Zhirui+Chen%2C+Fangzhi+Xu%2C+Kanzhi+Cheng%2C+Chang+Ma%2C+Zhangyue+Yin%2C+Jianing+Wang%2C+Chengcheng+Han%2C+Renyu+Zhu%2C+Shuai+Yuan%2C+et+al%2E+2024%2E+A+survey+of+neural+code+intelligence%3A+Paradigms%2C+advances+a)
43. AoPSWiki Team. 2025. Aime problems and solutions. [find](https://scholar.google.com/scholar?q=AoPSWiki+Team%2E+2025%2E+Aime+problems+and+solutions%2E)
44. Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. 2025. Atom of thoughts for markov llm test-time scaling. arXiv preprint arXiv:2502.12018. — [arXiv:2502.12018](https://arxiv.org/abs/2502.12018) [find](https://scholar.google.com/scholar?q=Fengwei+Teng%2C+Zhaoyang+Yu%2C+Quan+Shi%2C+Jiayi+Zhang%2C+Chenglin+Wu%2C+and+Yuyu+Luo%2E+2025%2E+Atom+of+thoughts+for+markov+llm+test%2Dtime+scaling%2E+arXiv+preprint+arXiv%3A2502%2E12018%2E+%E2%80%94+%5BarXiv%3A2502%2E12018%5D%28https%3A%2F%2Farxi)
45. Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. Position: [Will we run out of data? limits of llm scaling based on human-generated data](https://md.research.elamur.ai/free/en/papers/villalobos-2022-will-run-out-data-limits/). In Forty-first International Conference on Machine Learning.
46. Jian Wang, Boyan Zhu, Chak Tou Leong, Yongqi Li, and Wenjie Li. 2025a. Scaling over scaling: Exploring test-time scaling pareto in large reasoning models. arXiv preprint arXiv:2505.20522. — [arXiv:2505.20522](https://arxiv.org/abs/2505.20522) [find](https://scholar.google.com/scholar?q=Jian+Wang%2C+Boyan+Zhu%2C+Chak+Tou+Leong%2C+Yongqi+Li%2C+and+Wenjie+Li%2E+2025a%2E+Scaling+over+scaling%3A+Exploring+test%2Dtime+scaling+pareto+in+large+reasoning+models%2E+arXiv+preprint+arXiv%3A2505%2E20522%2E+%E2%80%94+%5BarXiv%3A250)
47. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. 2024a. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). Advances in Neural Information Processing Systems, 37:95266– 95290.
48. Yue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang, Xingyu Chen, Zhiwei He, Linfeng Song, Dian Yu, Juntao Li, Zhuosheng Zhang, et al. 2025b. Thoughts are all over the place: On the underthinking of o1-like llms. arXiv preprint arXiv:2501.18585. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Qiuzhi+Liu%2C+Jiahao+Xu%2C+Tian+Liang%2C+Xingyu+Chen%2C+Zhiwei+He%2C+Linfeng+Song%2C+Dian+Yu%2C+Juntao+Li%2C+Zhuosheng+Zhang%2C+et+al%2E+2025b%2E+Thoughts+are+all+over+the+place%3A+On+the+underthinking+of+o1%2Dlike+l)
49. Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. 2024b. [Charxiv: Charting gaps in realistic chart understanding in multimodal llms](https://md.research.elamur.ai/free/en/papers/wang-2024-charxiv-charting-gaps-realistic-chart/). Advances in Neural Information Processing Systems, 37:113569–113697.
50. Guojun Wu. 2025. It’s not that simple. an analysis of simple test-time scaling. arXiv preprint arXiv:2507.14419. — [arXiv:2507.14419](https://arxiv.org/abs/2507.14419) [find](https://scholar.google.com/scholar?q=Guojun+Wu%2E+2025%2E+It%E2%80%99s+not+that+simple%2E+an+analysis+of+simple+test%2Dtime+scaling%2E+arXiv+preprint+arXiv%3A2507%2E14419%2E+%E2%80%94+%5BarXiv%3A2507%2E14419%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2507%2E14419%29)
51. Yangzhen Wu, Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. 2025. [Inference scaling laws: An empirical analysis of compute-optimal inference for problem-solving with language models](https://md.research.elamur.ai/free/en/papers/wu-2024-inference-scaling-laws-empirical-analysis/). In The Thirteenth International Conference on Learning Representations. OpenReview.
52. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, Felix Juefei-Xu, Foutse Khomh, Osamu Yoshie, Qingyu Chen, Douglas Teodoro, Na… — [MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation](https://md.research.elamur.ai/free/en/papers/xuan-2025-mmlu-prox-multilingual-benchmark-advanced/)
53. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
54. Zhangyue Yin, Qiushi Sun, Zhiyuan Zeng, Qinyuan Cheng, Xipeng Qiu, and Xuanjing Huang. 2025. Dynamic and generalizable process reward modeling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4203–4233, Vienna, Austria. Associ… [find](https://scholar.google.com/scholar?q=Zhangyue+Yin%2C+Qiushi+Sun%2C+Zhiyuan+Zeng%2C+Qinyuan+Cheng%2C+Xipeng+Qiu%2C+and+Xuanjing+Huang%2E+2025%2E+Dynamic+and+generalizable+process+reward+modeling%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Associa)
55. Zhaojian Yu, Yinghao Wu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. 2025. Z1: Efficient test-time scaling with code. arXiv preprint arXiv:2504.00810. [find](https://scholar.google.com/scholar?q=Zhaojian+Yu%2C+Yinghao+Wu%2C+Yilun+Zhao%2C+Arman+Cohan%2C+and+Xiao%2DPing+Zhang%2E+2025%2E+Z1%3A+Efficient+test%2Dtime+scaling+with+code%2E+arXiv+preprint+arXiv%3A2504%2E00810%2E)
56. Xiang Yue, Yuansheng Ni, Tianyu Zheng, Kai Zhang, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. [Mmmu: A massive multi-disc…](https://md.research.elamur.ai/free/en/papers/yue-2023-mmmu-massive-multi-discipline-multimodal/)
57. Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Bo Wang, Shimin Li, Yunhua Zhou, Qipeng Guo, Xuanjing Huang, and Xipeng Qiu. 2024. Scaling of search and learning: A roadmap to reproduce o1 from reinforcement learning perspective. arXiv preprint arXiv:2412.14135. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Qinyuan+Cheng%2C+Zhangyue+Yin%2C+Bo+Wang%2C+Shimin+Li%2C+Yunhua+Zhou%2C+Qipeng+Guo%2C+Xuanjing+Huang%2C+and+Xipeng+Qiu%2E+2024%2E+Scaling+of+search+and+learning%3A+A+roadmap+to+reproduce+o1+from+reinforceme)
58. Zhiyuan Zeng, Qinyuan Cheng, Zhangyue Yin, Yunhua Zhou, and Xipeng Qiu. 2025. [Revisiting the test-time scaling of o1-like models: Do they truly possess test-time scaling capabilities?](https://md.research.elamur.ai/free/en/papers/zeng-2025-revisiting-test-time-scaling-o1/) In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
59. Ming Zhang, Kexin Tan, Yueyuan Huang, Yujiong Shen, Chunchun Ma, Li Ju, Xinran Zhang, Yuhui Wang, Wenqing Jing, Jingyi Deng, et al. 2026. Opennovelty: An llm-powered agentic system for verifiable scholarly novelty assessment. arXiv preprint arXiv:2601.01576. — [arXiv:2601.01576](https://arxiv.org/abs/2601.01576) [find](https://scholar.google.com/scholar?q=Ming+Zhang%2C+Kexin+Tan%2C+Yueyuan+Huang%2C+Yujiong+Shen%2C+Chunchun+Ma%2C+Li+Ju%2C+Xinran+Zhang%2C+Yuhui+Wang%2C+Wenqing+Jing%2C+Jingyi+Deng%2C+et+al%2E+2026%2E+Opennovelty%3A+An+llm%2Dpowered+agentic+system+for+verifiable+scho)
60. Ming Zhang, Yue Zhang, Shichun Liu, Haipeng Yuan, Junzhe Wang, Yurui Dong, Jingyi Deng, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. Llmeval-2. [find](https://scholar.google.com/scholar?q=Ming+Zhang%2C+Yue+Zhang%2C+Shichun+Liu%2C+Haipeng+Yuan%2C+Junzhe+Wang%2C+Yurui+Dong%2C+Jingyi+Deng%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2023%2E+Llmeval%2D2%2E)
61. Qiyuan Zhang, Fuyuan Lyu, Zexu Sun, Lei Wang, Weixu Zhang, Wenyue Hua, Haolun Wu, Zhihan Guo, Yufei Wang, Niklas Muennighoff, et al. 2025. [A survey on test-time scaling in large language models: What, how, where, and how well?](https://md.research.elamur.ai/free/en/papers/zhang-2025-survey-test-time-scaling-large/) arXiv preprint arXiv:2503.24235.
62. Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Llmeval: A preliminary study on how to evaluate large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19615–19622. [find](https://scholar.google.com/scholar?q=Yue+Zhang%2C+Ming+Zhang%2C+Haipeng+Yuan%2C+Shichun+Liu%2C+Yongyao+Shi%2C+Tao+Gui%2C+Qi+Zhang%2C+and+Xuanjing+Huang%2E+2024%2E+Llmeval%3A+A+preliminary+study+on+how+to+evaluate+large+language+models%2E+In+Proceedings+of+the)
63. King Zhu, Hanhao Li, Siwei Wu, Tianshun Xing, Dehua Ma, Xiangru Tang, Minghao Liu, Jian Yang, Jiaheng Liu, Yuchen Eleanor Jiang, et al. 2025. Scaling test-time compute for llm agents. arXiv preprint arXiv:2506.12928. — [arXiv:2506.12928](https://arxiv.org/abs/2506.12928) [find](https://scholar.google.com/scholar?q=King+Zhu%2C+Hanhao+Li%2C+Siwei+Wu%2C+Tianshun+Xing%2C+Dehua+Ma%2C+Xiangru+Tang%2C+Minghao+Liu%2C+Jian+Yang%2C+Jiaheng+Liu%2C+Yuchen+Eleanor+Jiang%2C+et+al%2E+2025%2E+Scaling+test%2Dtime+compute+for+llm+agents%2E+arXiv+preprint+a)

*Updated 2 November 2026*