---
title: "PRBench — paper by Akyürek et al. (2026) — summary and ideas"
description: "The open release gives researchers a large, transparent tool for measuring professional reasoning. 41 references, cited by 7 papers here."
authors: ["Afra Feyza Akyürek", "Advait Gosai", "Chen Bo Calvin Zhang", "Vipul Gupta", "Jaehwan Jeong", "Anisha Gunjal", "Tahseen Rabbani", "Maria Mazzone", "David Randolph IV", "Mohammad Mahmoudi Meymand", "Gurshaan Chattha", "Paula Rodriguez", "Diego A. Mares Buendia", "Pavit Singh", "Michael Liu", "Subodh Chawla", "Peter Cline", "Lucy Ogaz", "Ernesto Gabriel Hernández Montoya", "Zihao Wang", "Pavi Bhatter", "Marcos Ayestaran", "Bing Liu", "Yunzhong He"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1958/"
doi: "10.18653/v1/2026.acl-long.1958"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/akyurek-2026-prbench-large-scale-expert-rubrics/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/akyurek-2026-prbench-large-scale-expert-rubrics/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/) / PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

# PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning

**Authors:** [Afra Feyza Akyürek](https://md.research.elamur.ai/free/en/authors/afra-feyza-akyurek/), Advait Gosai, [Chen Bo Calvin Zhang](https://md.research.elamur.ai/free/en/authors/chen-bo-calvin-zhang/), [Vipul Gupta](https://md.research.elamur.ai/free/en/authors/vipul-gupta/), Jaehwan Jeong, [Anisha Gunjal](https://md.research.elamur.ai/free/en/authors/anisha-gunjal/), Tahseen Rabbani, Maria Mazzone, David Randolph IV, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego A. Mares Buendia, Pavit Singh, Michael Liu, Subodh Chawla, Peter Cline, Lucy Ogaz, Ernesto Gabriel Hernández Montoya, [Zihao Wang](https://md.research.elamur.ai/free/en/authors/zihao-wang/), Pavi Bhatter, Marcos Ayestaran, [Bing Liu](https://md.research.elamur.ai/free/en/authors/bing-liu/) and Yunzhong He · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1958/) · [PDF](https://aclanthology.org/2026.acl-long.1958.pdf) · [DOI 10.18653/v1/2026.acl-long.1958](https://doi.org/10.18653/v1/2026.acl-long.1958) · [OpenAlex](https://openalex.org/W4416368663)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [7](https://md.research.elamur.ai/free/en/papers/akyurek-2026-prbench-large-scale-expert-rubrics/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most LLM benchmarks reward short, checkable answers in math, science and coding. They reveal little about open-ended, high-stakes professional work. Preference arenas, the usual option for open-ended evaluation, produce noisy scores without expert grounding. The Scale AI authors address this gap with PRBench, a set of 1,100 realistic Finance and Law tasks. The tasks were written by 182 vetted professionals: lawyers hold JDs, and finance contributors have a Master's, a CFA or 6+ years of practice. Each task draws on the expert's own workflow or on questions from clients. Every task has an expert-built rubric of 10–30 weighted criteria. A second expert reviews each rubric, and automated quality checks follow, so an LLM judge can grade responses interpretably. Across more than 20 proprietary and open models, even the strongest score only about 0.40 on the Hard subsets. Models handle instruction following fairly well but often make wrong judgments or reason opaquely and incompletely. The open release gives researchers a large, transparent tool for measuring professional reasoning.

## Key points
- **Scale and coverage:** 600 Finance and 500 Legal tasks carry 18,711 rubric criteria (9,879 Finance, 8,832 Legal). About 30% of conversations are multi-turn. The tasks span 13 Finance and 12 Legal topics, 114 countries and 47 US jurisdictions.
- **Hard subsets:** 300 Finance and 250 Legal tasks were selected by low average model scores. The best results are 0.40 (Finance) and 0.41 (Legal), averaged over three runs with 95% confidence intervals.
- **Conversation construction:** Experts wrote the user turns in English. Assistant turns were sampled from GPT OSS 20B, Mistral Medium or DeepSeek R1. A private held-out set is kept to detect future contamination.
- **Resources:** The data is available at Hugging Face `ScaleAI/PRBench`, and the evaluation code at GitHub `scaleapi/PRBench`. Prompts are also annotated for economic impact and decision type.
- **Limitations:** LLM judges can over-credit answers that only look correct. Tasks are English-only and exclude tool use (databases, spreadsheets).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,772 works · [🤝 Political Science and International Relations](https://md.research.elamur.ai/free/en/shelves/social-sciences/political-science-and-international-relations/) · 182 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph IV, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego A. Mares Buendia, Pavit Singh, Michael Liu, Subodh Chawla, Peter Cline, Lucy Ogaz, Ernesto Gabriel Hernández Montoya, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu and Yunzhong He. 2026. PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{akyurek2026prbench,
  title = {{PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning}},
  author = {Afra Feyza Akyürek, Advait Gosai, Chen Bo Calvin Zhang, Vipul Gupta, Jaehwan Jeong, Anisha Gunjal, Tahseen Rabbani, Maria Mazzone, David Randolph IV, Mohammad Mahmoudi Meymand, Gurshaan Chattha, Paula Rodriguez, Diego A. Mares Buendia, Pavit Singh, Michael Liu, Subodh Chawla, Peter Cline, Lucy Ogaz, Ernesto Gabriel Hernández Montoya, Zihao Wang, Pavi Bhatter, Marcos Ayestaran, Bing Liu and Yunzhong He},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1958},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1958}
}
```

- [← previous New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Models](https://md.research.elamur.ai/free/en/papers/wunderle-2026-new-encoders-german-trained-scratch/)
- [next → ProgressLM: Towards Progress Reasoning in Vision-Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2026-progresslm-progress-reasoning-vision-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/2/)

## References

41 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ruth Appel, Peter McCrory, Alex Tamkin, Michael Stern, Miles McCain, and Tyler Neylon. 2025. Anthropic economic index report: Uneven geographic and enterprise ai adoption. Anthropic Research. [find](https://scholar.google.com/scholar?q=Ruth+Appel%2C+Peter+McCrory%2C+Alex+Tamkin%2C+Michael+Stern%2C+Miles+McCain%2C+and+Tyler+Neylon%2E+2025%2E+Anthropic+economic+index+report%3A+Uneven+geographic+and+enterprise+ai+adoption%2E+Anthropic+Research%2E)
2. Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. 2025. [Healthbench: Evaluating large language models towards improved human health](https://md.research.elamur.ai/free/en/papers/arora-2025-healthbench-evaluating-large-language-models/). Preprint…
3. Aaron Chatterji, Thomas Cunningham, David J Deming, Zoe Hitzig, Christopher Ong, Carl Yan Shan, and Kevin Wadman. 2025. How people use chatgpt. Technical report, National Bureau of Economic Research. [find](https://scholar.google.com/scholar?q=Aaron+Chatterji%2C+Thomas+Cunningham%2C+David+J+Deming%2C+Zoe+Hitzig%2C+Christopher+Ong%2C+Carl+Yan+Shan%2C+and+Kevin+Wadman%2E+2025%2E+How+people+use+chatgpt%2E+Technical+report%2C+National+Bureau+of+Economic+Research%2E)
4. Zhiyu Chen, SHIYANG LI, Charese Smiley, Zhiqiang Ma, Sameena Shah, and William Yang Wang. 2022. [Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering](https://md.research.elamur.ai/free/en/papers/chen-2022-convfinqa-exploring-chain-numerical-reasoning/). In Conference on Empirical Methods in Natural Language Processing.
5. Odysseas S. Chlapanis, Dimitrios Galanis, Nikolaos Aletras, and Ion Androutsopoulos. 2025. GreekBar-Bench: A challenging benchmark for free-text legal reasoning and citations. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 25099–25119, Suzhou, China. Association for… [find](https://scholar.google.com/scholar?q=Odysseas+S%2E+Chlapanis%2C+Dimitrios+Galanis%2C+Nikolaos+Aletras%2C+and+Ion+Androutsopoulos%2E+2025%2E+GreekBar%2DBench%3A+A+challenging+benchmark+for+free%2Dtext+legal+reasoning+and+citations%2E+In+Findings+of+the+Assoc)
6. Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. 2025. Arc-agi-2: A new challenge for frontier ai reasoning systems. arXiv preprint arXiv:2505.11831. [find](https://scholar.google.com/scholar?q=Francois+Chollet%2C+Mike+Knoop%2C+Gregory+Kamradt%2C+Bryan+Landers%2C+and+Henry+Pinkard%2E+2025%2E+Arc%2Dagi%2D2%3A+A+new+challenge+for+frontier+ai+reasoning+systems%2E+arXiv+preprint+arXiv%3A2505%2E11831%2E)
7. Eva Eigner and Thorsten Händler. 2024. Determinants of llm-assisted decision-making. arXiv preprint arXiv:2402.17385. — [arXiv:2402.17385](https://arxiv.org/abs/2402.17385) [find](https://scholar.google.com/scholar?q=Eva+Eigner+and+Thorsten+H%C3%A4ndler%2E+2024%2E+Determinants+of+llm%2Dassisted+decision%2Dmaking%2E+arXiv+preprint+arXiv%3A2402%2E17385%2E+%E2%80%94+%5BarXiv%3A2402%2E17385%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E17385%29)
8. Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstruwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, Daniel Brunner, Markus Leippold, Mrinmaya Sachan, Alexander Stremitzer, Christoph Engel, Elliott Ash, and Joel Niklaus. 2025. Lexam: Benchmarking legal rea… [find](https://scholar.google.com/scholar?q=Yu+Fan%2C+Jingwei+Ni%2C+Jakob+Merane%2C+Yang+Tian%2C+Yoan+Hermstruwer%2C+Yinya+Huang%2C+Mubashara+Akhtar%2C+Etienne+Salimbeni%2C+Florian+Geering%2C+Oliver+Dreyer%2C+Daniel+Brunner%2C+Markus+Leippold%2C+Mrinmaya+Sachan%2C+Alexa)
9. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Re, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21 ot… — [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/)
10. Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. 2025. Rubrics as rewards: Reinforcement learning beyond verifiable domains. Preprint, arXiv:2507.17746. [find](https://scholar.google.com/scholar?q=Anisha+Gunjal%2C+Anthony+Wang%2C+Elaine+Lau%2C+Vaskar+Nath%2C+Yunzhong+He%2C+Bing+Liu%2C+and+Sean+Hendryx%2E+2025%2E+Rubrics+as+rewards%3A+Reinforcement+learning+beyond+verifiable+domains%2E+Preprint%2C+arXiv%3A2507%2E17746%2E)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In Proceedings of the International Conference on Learning Representations (ICLR).
12. Luis Hillebrand, Sebastian Raisch, and Jonathan Schad. 2025. Managing with artificial intelligence: An integrative framework. Academy of Management Annals, 19(1):343–375. [find](https://scholar.google.com/scholar?q=Luis+Hillebrand%2C+Sebastian+Raisch%2C+and+Jonathan+Schad%2E+2025%2E+Managing+with+artificial+intelligence%3A+An+integrative+framework%2E+Academy+of+Management+Annals%2C+19%281%29%3A343%E2%80%93375%2E)
13. Zenan Huang, Yihong Zhuang, Guoshan Lu, Zeyu Qin, Haokai Xu, Tianyu Zhao, Ru Peng, Jiaqi Hu, Zhanming Shen, Xiaomeng Hu, Xijun Gu, Peiyi Tu, Jiaxin Liu, Wenyu Chen, Yuzhuo Fu, Zhiting Fan, Yanmei Gu, Yuanyuan Wang, Zhengkai Yang, and 2 others. 2025. Reinforcement learning with rubric anchors. Prepri… — [arXiv:2508.12790](https://arxiv.org/abs/2508.12790) [find](https://scholar.google.com/scholar?q=Zenan+Huang%2C+Yihong+Zhuang%2C+Guoshan+Lu%2C+Zeyu+Qin%2C+Haokai+Xu%2C+Tianyu+Zhao%2C+Ru+Peng%2C+Jiaqi+Hu%2C+Zhanming+Shen%2C+Xiaomeng+Hu%2C+Xijun+Gu%2C+Peiyi+Tu%2C+Jiaxin+Liu%2C+Wenyu+Chen%2C+Yuzhuo+Fu%2C+Zhiting+Fan%2C+Yanmei+Gu%2C+)
14. Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. Financebench: A new benchmark for financial question answering. ArXiv, abs/2311.11944. — [arXiv:2311.11944](https://arxiv.org/abs/2311.11944) [find](https://scholar.google.com/scholar?q=Pranab+Islam%2C+Anand+Kannappan%2C+Douwe+Kiela%2C+Rebecca+Qian%2C+Nino+Scherrer%2C+and+Bertie+Vidgen%2E+2023%2E+Financebench%3A+A+new+benchmark+for+financial+question+answering%2E+ArXiv%2C+abs%2F2311%2E11944%2E+%E2%80%94+%5BarXiv%3A2311%2E1)
15. Mohsen Khosravi, Zahra Zare, Seyyed M. Mojtabaeian, and Reyhane Izadi. Artificial intelligence and decision-making in healthcare: A thematic analysis of a systematic review of reviews. Health Services Research and Managerial Epidemiology, 11. [find](https://scholar.google.com/scholar?q=Mohsen+Khosravi%2C+Zahra+Zare%2C+Seyyed+M%2E+Mojtabaeian%2C+and+Reyhane+Izadi%2E+Artificial+intelligence+and+decision%2Dmaking+in+healthcare%3A+A+thematic+analysis+of+a+systematic+review+of+reviews%2E+Health+Services)
16. Sihyun Kim, Sangyoon Yi, and Sung-Pil Park. 2025. Prioritizing challenges in ai adoption for the legal domain: A systematic review and expert-driven AHP analysis. PLOS ONE, 20(6):e0326028. [find](https://scholar.google.com/scholar?q=Sihyun+Kim%2C+Sangyoon+Yi%2C+and+Sung%2DPil+Park%2E+2025%2E+Prioritizing+challenges+in+ai+adoption+for+the+legal+domain%3A+A+systematic+review+and+expert%2Ddriven+AHP+analysis%2E+PLOS+ONE%2C+20%286%29%3Ae0326028%2E)
17. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
18. LMArena. 2025. Arena expert and occupational categories. LMArena News. Blog post. [find](https://scholar.google.com/scholar?q=LMArena%2E+2025%2E+Arena+expert+and+occupational+categories%2E+LMArena+News%2E+Blog+post%2E)
19. LMArena Team. 2025. Arena expert and occupational categories. [https://news.lmarena.ai/](https://news.lmarena.ai/) arena-expert/. Accessed: 2025-11-11. [↗](https://news.lmarena.ai/)
20. Mantas Mazeika, Alice Gatti, Cristina Menghini, Udari Madhushani Sehwag, Shivam Singhal, Yury Orlovskiy, Steven Basart, Manasi Sharma, Denis Peskoff, Elaine Lau, Jaehyuk Lim, Lachlan Carroll, Alice Blair, Vinaya Sivakumar, Sumana Basu, Brad Kenstler, Yuntao Ma, Julian Michael, Xiaoke Li, and 28 othe… — [arXiv:2510.26787](https://arxiv.org/abs/2510.26787) [find](https://scholar.google.com/scholar?q=Mantas+Mazeika%2C+Alice+Gatti%2C+Cristina+Menghini%2C+Udari+Madhushani+Sehwag%2C+Shivam+Singhal%2C+Yury+Orlovskiy%2C+Steven+Basart%2C+Manasi+Sharma%2C+Denis+Peskoff%2C+Elaine+Lau%2C+Jaehyuk+Lim%2C+Lachlan+Carroll%2C+Alice+Bl)
21. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. 2025. Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering? Preprint, arXiv:2502.12115. nof1.ai. 2025. Alpha arena. [https://nof1.ai/](https://nof1.ai/). Accessed: 2025-11-11. [↗](https://nof1.ai/)
22. Melissa Pan, Negar Arabzadeh, Riccardo Cogo, Yuxuan Zhu, Alexander Xiong, Lakshya Agrawal, Huanzhi Mao, Emma Shen, Sid Pallerla, Liana Patel, Shu Liu, Tianneng Shi, Xiaoyuan Liu, Jared Davis, Emmanuele Lacavalla, Alessandro Basile, Shuyi Yang, Paul Castro, Daniel Kang, and 6 others. 2025. Measuring… — [arXiv:2512.04123](https://arxiv.org/abs/2512.04123) [find](https://scholar.google.com/scholar?q=Melissa+Pan%2C+Negar+Arabzadeh%2C+Riccardo+Cogo%2C+Yuxuan+Zhu%2C+Alexander+Xiong%2C+Lakshya+Agrawal%2C+Huanzhi+Mao%2C+Emma+Shen%2C+Sid+Pallerla%2C+Liana+Patel%2C+Shu+Liu%2C+Tianneng+Shi%2C+Xiaoyuan+Liu%2C+Jared+Davis%2C+Emmanuel)
23. Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2025. Limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI… [find](https://scholar.google.com/scholar?q=Annalisa+Szymanski%2C+Noah+Ziems%2C+Heather+A%2E+Eicher%2DMiller%2C+Toby+Jia%2DJun+Li%2C+Meng+Jiang%2C+and+Ronald+A%2E+Metoyer%2E+2025%2E+Limitations+of+the+llm%2Das%2Da%2Djudge+approach+for+evaluating+llm+outputs+in+expert+know)
24. Tejal Patwardhan, Rachel Dias, Elizabeth Proehl, Grace Kim, Michele Wang, Olivia Watkins, Simón Posada Fishman, Marwan Aljubeh, Phoebe Thacker, Laurance Fauconnet, Natalie S. Kim, Patrick Chao, Samuel Miserendino, Gildas Chabot, David Li, Michael Sharman, Alexandra Barr, Amelia Glaese, and Jerry Two… [find](https://scholar.google.com/scholar?q=Tejal+Patwardhan%2C+Rachel+Dias%2C+Elizabeth+Proehl%2C+Grace+Kim%2C+Michele+Wang%2C+Olivia+Watkins%2C+Sim%C3%B3n+Posada+Fishman%2C+Marwan+Aljubeh%2C+Phoebe+Thacker%2C+Laurance+Fauconnet%2C+Natalie+S%2E+Kim%2C+Patrick+Chao%2C+Samuel)
25. Zichen Tang, Haihong E, Ziyan Ma, Haoyang He, Jiacheng Liu, Zhongjun Yang, Zihua Rong, Rongjin Li, Kun Ji, Qing Huang, Xinyang Hu, Yang Liu, and Qianhe Zheng. 2025. [FinanceReasoning: Benchmarking financial numerical reasoning more credible, comprehensive and challenging](https://md.research.elamur.ai/free/en/papers/tang-2025-financereasoning-benchmarking-financial-numerical-reasoning/). In Proceedings of the 63rd A…
26. Julio Pereyra, Elizabeth Lebens, Matthew Guillod, Laura Toulme, Cameron MacGregor, David Mur-dter, Karl de la Roche, Emilie McConnachie, Jeremy Pushkin, Rina Kim, Aaron Chan, Jenny Pan, Boling Yang, Nan Wu, Niko Grupen, Lauren Oh, Aatish Nayak, and Gabriel Pereyra. 2024. Introducing biglaw bench. ht… [find](https://scholar.google.com/scholar?q=Julio+Pereyra%2C+Elizabeth+Lebens%2C+Matthew+Guillod%2C+Laura+Toulme%2C+Cameron+MacGregor%2C+David+Mur%2Ddter%2C+Karl+de+la+Roche%2C+Emilie+McConnachie%2C+Jeremy+Pushkin%2C+Rina+Kim%2C+Aaron+Chan%2C+Jenny+Pan%2C+Boling+Yang%2C+N)
27. Long Phan, Alice Gatti, Nathaniel Li, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dan Hendrycks, Ziwen Han, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, and 1081 others. 2026. A benchmark of e… [find](https://scholar.google.com/scholar?q=Long+Phan%2C+Alice+Gatti%2C+Nathaniel+Li%2C+Adam+Khoja%2C+Ryan+Kim%2C+Richard+Ren%2C+Jason+Hausenloy%2C+Oliver+Zhang%2C+Mantas+Mazeika%2C+Dan+Hendrycks%2C+Ziwen+Han%2C+Josephina+Hu%2C+Hugh+Zhang%2C+Chen+Bo+Calvin+Zhang%2C+Mohame)
28. Inc. Vals AI. 2025. Corpfin (v2): A private benchmark evaluating understanding of long-context credit agreements. Online benchmark. Accessed on April 20, 2026. [find](https://scholar.google.com/scholar?q=Inc%2E+Vals+AI%2E+2025%2E+Corpfin+%28v2%29%3A+A+private+benchmark+evaluating+understanding+of+long%2Dcontext+credit+agreements%2E+Online+benchmark%2E+Accessed+on+April+20%2C+2026%2E)
29. Bertie Vidgen, Abby Fennelly, Evan Pinnix, Chirag Mahapatra, Zach Richards, Austin Bridges, Calix Huang, Ben Hunsberger, Fez Zafar, Brendan Foody, Dominic Barton, Cass R. Sunstein, Eric Topol, and Osvald Nitski. 2025. The ai productivity index (apex). Preprint, arXiv:2509.25721. — [arXiv:2509.25721](https://arxiv.org/abs/2509.25721) [find](https://scholar.google.com/scholar?q=Bertie+Vidgen%2C+Abby+Fennelly%2C+Evan+Pinnix%2C+Chirag+Mahapatra%2C+Zach+Richards%2C+Austin+Bridges%2C+Calix+Huang%2C+Ben+Hunsberger%2C+Fez+Zafar%2C+Brendan+Foody%2C+Dominic+Barton%2C+Cass+R%2E+Sunstein%2C+Eric+Topol%2C+and+Osv)
30. Darko B. Vuković, Senanu Dekpo-Adza, and Stefana Matović. 2025. AI integration in financial services: A systematic review of trends and regulatory challenges. Humanities and Social Sciences Communications, 12(1):562. [find](https://scholar.google.com/scholar?q=Darko+B%2E+Vukovic%CC%81%2C+Senanu+Dekpo%2DAdza%2C+and+Stefana+Matovic%CC%81%2E+2025%2E+AI+integration+in+financial+services%3A+A+systematic+review+of+trends+and+regulatory+challenges%2E+Humanities+and+Social+Sciences+Communic)
31. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
32. Zhilin Wang, Jaehun Jung, Ximing Lu, Shizhe Diao, Ellie Evans, Jiaqi Zeng, Pavlo Molchanov, Yejin Choi, Jan Kautz, and Yi Dong. 2025. Profbench: Multi-domain rubrics requiring professional knowledge to answer and judge. Preprint, arXiv:2510.18941. — [arXiv:2510.18941](https://arxiv.org/abs/2510.18941) [find](https://scholar.google.com/scholar?q=Zhilin+Wang%2C+Jaehun+Jung%2C+Ximing+Lu%2C+Shizhe+Diao%2C+Ellie+Evans%2C+Jiaqi+Zeng%2C+Pavlo+Molchanov%2C+Yejin+Choi%2C+Jan+Kautz%2C+and+Yi+Dong%2E+2025%2E+Profbench%3A+Multi%2Ddomain+rubrics+requiring+professional+knowledge+t)
33. MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, and Afra Feyza Akyürek. 2025. Online rubrics elicitation from pairwise comparisons. Preprint, arXiv:2510.07284. — [arXiv:2510.07284](https://arxiv.org/abs/2510.07284) [find](https://scholar.google.com/scholar?q=MohammadHossein+Rezaei%2C+Robert+Vacareanu%2C+Zihao+Wang%2C+Clinton+Wang%2C+Bing+Liu%2C+Yunzhong+He%2C+and+Afra+Feyza+Aky%C3%BCrek%2E+2025%2E+Online+rubrics+elicitation+from+pairwise+comparisons%2E+Preprint%2C+arXiv%3A2510%2E0728)
34. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025. Browsecomp: A simple yet challenging benchmark for browsing agents. Preprint, arXiv:2504.12516. [find](https://scholar.google.com/scholar?q=Jason+Wei%2C+Zhiqing+Sun%2C+Spencer+Papay%2C+Scott+McKinney%2C+Jeffrey+Han%2C+Isa+Fulford%2C+Hyung+Won+Chung%2C+Alex+Tachard+Passos%2C+William+Fedus%2C+and+Amelia+Glaese%2E+2025%2E+Browsecomp%3A+A+simple+yet+challenging+benc)
35. Nikki Rojas. 2024. Does ai help humans make better decisions? Harvard Gazette. [find](https://scholar.google.com/scholar?q=Nikki+Rojas%2E+2024%2E+Does+ai+help+humans+make+better+decisions%3F+Harvard+Gazette%2E)
36. Scale AI. 2025. Seal showdown: Technical report. Technical report, Scale AI. Preliminary results and methodology for the SEAL Showdown leaderboard. [find](https://scholar.google.com/scholar?q=Scale+AI%2E+2025%2E+Seal+showdown%3A+Technical+report%2E+Technical+report%2C+Scale+AI%2E+Preliminary+results+and+methodology+for+the+SEAL+Showdown+leaderboard%2E)
37. Ved Sirdeshmukh, Kaustubh Deshpande, Johannes Mols, Lifeng Jin, Ed-Yeremai Cardona, Dean Lee, Jeremy Kritz, Willow Primack, Summer Yue, and Chen Xing. 2025. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms. Preprint, arXiv:2501.17399. [find](https://scholar.google.com/scholar?q=Ved+Sirdeshmukh%2C+Kaustubh+Deshpande%2C+Johannes+Mols%2C+Lifeng+Jin%2C+Ed%2DYeremai+Cardona%2C+Dean+Lee%2C+Jeremy+Kritz%2C+Willow+Primack%2C+Summer+Yue%2C+and+Chen+Xing%2E+2025%2E+Multichallenge%3A+A+realistic+multi%2Dturn+conv)
38. Mark Steyvers and Aakriti Kumar. 2024. Three challenges for ai-assisted decision-making. Perspectives on Psychological Science, 19(5):722–734. [find](https://scholar.google.com/scholar?q=Mark+Steyvers+and+Aakriti+Kumar%2E+2024%2E+Three+challenges+for+ai%2Dassisted+decision%2Dmaking%2E+Perspectives+on+Psychological+Science%2C+19%285%29%3A722%E2%80%93734%2E)
39. Jannik Zeiser. 2024. Owning decisions: Ai decision-support and the attributability-gap. Science and Engineering Ethics, 30(4):1–19. [find](https://scholar.google.com/scholar?q=Jannik+Zeiser%2E+2024%2E+Owning+decisions%3A+Ai+decision%2Dsupport+and+the+attributability%2Dgap%2E+Science+and+Engineering+Ethics%2C+30%284%29%3A1%E2%80%9319%2E)
40. Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin. 2025. Chasing the tail: Effective rubric-based reward modeling for large language model post-training. Preprint, arXiv:2509.21500. — [arXiv:2509.21500](https://arxiv.org/abs/2509.21500) [find](https://scholar.google.com/scholar?q=Junkai+Zhang%2C+Zihao+Wang%2C+Lin+Gui%2C+Swarnashree+Mysore+Sathyendra%2C+Jaehwan+Jeong%2C+Victor+Veitch%2C+Wei+Wang%2C+Yunzhong+He%2C+Bing+Liu%2C+and+Lifeng+Jin%2E+2025%2E+Chasing+the+tail%3A+Effective+rubric%2Dbased+reward+m)
41. Yilun Zhao, Hongjun Liu, Yitao Long, Rui Zhang, Chen Zhao, and Arman Cohan. 2024. [Financemath: Knowledge-intensive math reasoning in finance domains](https://md.research.elamur.ai/free/en/papers/zhou-2023-instruction-following-evaluation-large-language/). In Proceedings of the 62nd Annual Meeting of Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou…

*Updated 31 October 2026*