---
title: "Humanity's Last Exam — Phan 2025"
description: "Popular benchmarks such as MMLU are now close to saturated, so they no longer tell us much about how capable frontier LLMs have become."
authors: ["Long Phan", "Alice Gatti", "Ziwen Han", "Nathaniel Li", "Josephina Hu", "Hugh Zhang", "Chen Bo Calvin Zhang", "Mohamed Shaaban", "John Ling", "Sean Shi", "Michael Choi", "Anish Agrawal", "Arnav Chopra", "Adam Khoja", "Ryan Kim", "Richard Ren", "Jason Hausenloy", "Oliver Zhang", "Mantas Mazeika", "Dmitry Dodonov", "Tung Nguyen", "Jaeho Lee", "Daron Anderson", "Mikhail Doroshenko", "Alun Cennyth Stokes", "Mobeen Mahmood", "Oleksandr Pokutnyi", "Oleg Iskra", "Jessica P. Wang", "John-Clark Levin", "Mstyslav Kazakov", "Fiona Feng", "Steven Y. Feng", "Haoran Zhao", "Michael Yu", "Varun Gangal", "Chelsea Zou", "Zihan Wang", "Serguei Popov", "Robert Gerbicz", "Geoff Galgon", "Johannes Schmitt", "Will Yeadon", "Yongki Lee", "Scott Sauers", "Alvaro Sanchez", "Fabian Giska", "Marc Roth", "Søren Riis", "Saiteja Utpala", "Noah Burns", "Gashaw M. Goshu", "Mohinder Maheshbhai Naiya", "Chidozie Agu", "Zachary Giboney", "Antrell Cheatom", "Francesco Fournier-Facio", "Sarah-Jane Crowson", "Lennart Finke", "Zerui Cheng", "Jennifer Zampese", "Ryan G. Hoerr", "Mark Nandor", "Hyunwoo Park", "Tim Gehrunger", "Jiaqi Cai", "Ben McCarty", "Alexis C Garretson", "Edwin Taylor", "Damien Sileo", "Qiuyu Ren", "Usman Qazi", "Lianghui Li", "Jungbae Nam", "John B. Wydallis", "Pavel Arkhipov", "Jack Wei Lun Shi", "Aras Bacho", "Chris G. Willcocks", "Hangrui Cao", "Sumeet Motwani", "Emily de Oliveira Santos", "Johannes Veith", "Edward Vendrow", "Doru Cojoc", "Kengo Zenitani", "Joshua Robinson", "Longke Tang", "Yuqi Li", "Joshua Vendrow", "Natanael Wildner Fraga", "Vladyslav Kuchkin", "Andrey Pupasov Maksimov", "Pierre Marion", "Denis Efremov", "Jayson Lynch", "Kaiqu Liang", "Aleksandar Mikov", "Andrew Gritsevskiy", "Julien Guillod", "Gözdenur Demir", "Dakotah Martinez", "Ben Pageler", "Kevin Zhou", "Saeed Soori", "Ori Press", "Henry Tang", "Paolo Rissone", "Sean R. Green", "Lina Brüssel", "Moon Twayana", "Aymeric Dieuleveut", "Joseph Marvin Imperial", "Ameya Prabhu", "Jinzhou Yang", "Nick Crispino", "Arun Rao", "Dimitri Zvonkine", "Gabriel Loiseau", "Mikhail Kalinin", "Marco Lukas", "Ciprian Manolescu", "Nate Stambaugh", "Subrata Mishra", "Tad Hogg", "Carlo Bosio", "Brian P Coppola", "Julian Salazar", "Jaehyeok Jin", "Rafael Sayous", "Stefan Ivanov", "Philippe Schwaller", "Shaipranesh Senthilkuma", "Andres M Bran", "Andres Algaba", "Kelsey Van den Houte", "Lynn Van Der Sypt", "Brecht Verbeken", "David Noever", "Alexei Kopylov", "Benjamin Myklebust", "Bikun Li", "Lisa Schut", "Evgenii Zheltonozhskii", "Qiaochu Yuan", "Derek Lim", "Richard Stanley", "Tong Yang", "John Maar", "Julian Wykowski", "Martí Oller", "Anmol Sahu", "Cesare Giulio Ardito", "Yuzheng Hu", "Ariel Ghislain Kemogne Kamdoum", "Alvin Jin", "Tobias Garcia Vilchis", "Yuexuan Zu", "Martin Lackner", "James Koppel", "Gongbo Sun", "Daniil S. Antonenko", "Steffi Chern", "Bingchen Zhao", "Pierrot Arsene", "Joseph M Cavanagh", "Daofeng Li", "Jiawei Shen", "Donato Crisostomi", "Wenjin Zhang", "Ali Dehghan", "Sergey Ivanov", "David Perrella", "Nurdin Kaparov", "Allen Zang", "Ilia Sucholutsky", "Arina Kharlamova", "Daniil Orel", "Vladislav Poritski", "Shalev Ben-David", "Zachary Berger", "Parker Whitfill", "Michael Foster", "Daniel Munro", "Linh Ho", "Shankar Sivarajan", "Dan Bar Hava", "Aleksey Kuchkin", "David Holmes", "Alexandra Rodriguez-Romero", "Frank Sommerhage", "Anji Zhang", "Richard Moat", "Keith Schneider", "Zakayo Kazibwe", "Don Clarke", "Dae Hyun Kim", "Felipe Meneguitti Dias", "Sara Fish", "Veit Elser", "Tobias Kreiman", "Victor Efren Guadarrama Vilchis", "Immo Klose", "Ujjwala Anantheswaran", "Adam Zweiger", "Kaivalya Rawal", "Jeffery Li", "Jeremy Nguyen", "Nicolas Daans", "Haline Heidinger", "Maksim Radionov", "Václav Rozhoň", "Vincent Ginis", "Christian Stump", "Niv Cohen", "Rafał Poświata", "Josef Tkadlec", "Alan Goldfarb", "Chenguang Wang", "Piotr Padlewski", "Stanislaw Barzowski", "Kyle Montgomery", "Ryan Stendall", "Jamie Tucker-Foltz", "Jack Stade", "T. Ryan Rogers", "Tom Goertzen", "Declan Grabb", "Abhishek Shukla", "Alan Givré", "John Arnold Ambay", "Archan Sen", "Muhammad Fayez Aziz", "Mark H Inlow", "Hao He", "Ling Zhang", "Younesse Kaddar", "Ivar Ängquist", "Yanxu Chen", "Harrison K Wang", "Kalyan Ramakrishnan", "Elliott Thornley", "Antonio Terpin", "Hailey Schoelkopf", "Eric Zheng", "Avishy Carmi", "Ethan D. L. Brown", "Kelin Zhu", "Max Bartolo", "Richard Wheeler", "Martin Stehberger", "Peter Bradshaw", "JP Heimonen", "Kaustubh Sridhar", "Ido Akov", "Jennifer Sandlin", "Yury Makarychev", "Joanna Tam", "Hieu Hoang", "David M. Cunningham", "Vladimir Goryachev", "Demosthenes Patramanis", "Michael Krause", "Andrew Redenti", "David Aldous", "Jesyin Lai", "Shannon Coleman", "Jiangnan Xu", "Sangwon Lee", "Ilias Magoulas", "Sandy Zhao", "Ning Tang", "Michael K. Cohen", "Orr Paradise", "Jan Hendrik Kirchner", "Maksym Ovchynnikov", "Jason O. Matos", "Adithya Shenoy", "Michael Wang", "Yuzhou Nie", "Anna Sztyber-Betley", "Paolo Faraboschi", "Robin Riblet", "Jonathan Crozier", "Shiv Halasyamani", "Shreyas Verma", "Prashant Joshi", "Eli Meril", "Ziqiao Ma", "Jérémy Andréoletti", "Raghav Singhal", "Jacob Platnick", "Volodymyr Nevirkovets", "Luke Basler", "Alexander Ivanov", "Seri Khoury", "Nils Gustafsson", "Marco Piccardo", "Hamid Mostaghimi", "Qijia Chen", "Virendra Singh", "Tran Quoc Khánh", "Paul Rosu", "Hannah Szlyk", "Zachary Brown", "Himanshu Narayan", "Aline Menezes", "Jonathan Roberts", "William Alley", "Kunyang Sun", "Arkil Patel", "Max Lamparth", "Anka Reuel", "Linwei Xin", "Hanmeng Xu", "Jacob Loader", "Freddie Martin", "Zixuan Wang", "Andrea Achilleos", "Thomas Preu", "Tomek Korbak", "Ida Bosio", "Fereshteh Kazemi", "Ziye Chen", "Biró Bálint", "Eve J. Y. Lo", "Jiaqi Wang", "Maria Inês S. Nunes", "Jeremiah Milbauer", "M Saiful Bari", "Zihao Wang", "Behzad Ansarinejad", "Yewen Sun", "Stephane Durand", "Hossam Elgnainy", "Guillaume Douville", "Daniel Tordera", "George Balabanian", "Hew Wolff", "Lynna Kvistad", "Hsiaoyun Milliron", "Ahmad Sakor", "Murat Eron", "Andrew Favre D. O.", "Shailesh Shah", "Xiaoxiang Zhou", "Firuz Kamalov", "Sherwin Abdoli", "Tim Santens", "Shaul Barkan", "Allison Tee", "Robin Zhang", "Alessandro Tomasiello", "G. Bruno De Luca", "Shi-Zhuo Looi", "Vinh-Kha Le", "Noam Kolt", "Jiayi Pan", "Emma Rodman", "Jacob Drori", "Carl J Fossum", "Niklas Muennighoff", "Milind Jagota", "Ronak Pradeep", "Honglu Fan", "Jonathan Eicher", "Michael Chen", "Kushal Thaman", "William Merrill", "Moritz Firsching", "Carter Harris", "Stefan Ciobâcă", "Jason Gross", "Rohan Pandey", "Ilya Gusev", "Adam Jones", "Shashank Agnihotri", "Pavel Zhelnov", "Mohammadreza Mofayezi", "Alexander Piperski", "David K. Zhang", "Kostiantyn Dobarskyi", "Roman Leventov", "Ignat Soroko", "Joshua Duersch", "Vage Taamazyan", "Andrew Ho", "Wenjie Ma", "William Held", "Ruicheng Xian", "Armel Randy Zebaze", "Mohanad Mohamed", "Julian Noah Leser", "Michelle X Yuan", "Laila Yacar", "Johannes Lengler", "Katarzyna Olszewska", "Claudio Di Fratta", "Edson Oliveira", "Joseph W. Jackson", "Andy Zou", "Muthu Chidambaram", "Timothy Manik", "Hector Haffenden", "Dashiell Stander", "Ali Dasouqi", "Alexander Shen", "Bita Golshani", "David Stap", "Egor Kretov", "Mikalai Uzhou", "Alina Borisovna Zhidkovskaya", "Nick Winter", "Miguel Orbegozo Rodriguez", "Robert Lauff", "Dustin Wehr", "Colin Tang", "Zaki Hossain", "Shaun Phillips", "Fortuna Samuele", "Fredrik Ekström", "Angela Hammon", "Oam Patel", "Faraz Farhidi", "George Medley", "Forough Mohammadzadeh", "Madellene Peñaflor", "Haile Kassahun", "Alena Friedrich", "Rayner Hernandez Perez", "Daniel Pyda", "Taom Sakal", "Omkar Dhamane", "Ali Khajegili Mirabadi", "Eric Hallman", "Kenchi Okutsu", "Mike Battaglia", "Mohammad Maghsoudimehrabani", "Alon Amit", "Dave Hulbert", "Roberto Pereira", "Simon Weber", "Handoko", "Anton Peristyy", "Stephen Malina", "Mustafa Mehkary", "Rami Aly", "Frank Reidegeld", "Anna-Katharina Dick", "Cary Friday", "Mukhwinder Singh", "Hassan Shapourian", "Wanyoung Kim", "Mariana Costa", "Hubeyb Gurdogan", "Harsh Kumar", "Chiara Ceconello", "Chao Zhuang", "Haon Park", "Micah Carroll", "Andrew R. Tawfeek", "Stefan Steinerberger", "Daattavya Aggarwal", "Michael Kirchhof", "Linjie Dai", "Evan Kim", "Johan Ferret", "Jainam Shah", "Yuzhou Wang", "Minghao Yan", "Krzysztof Burdzy", "Lixin Zhang", "Antonio Franca", "Diana T. Pham", "Kang Yong Loh", "Joshua Robinson", "Abram Jackson", "Paolo Giordano", "Philipp Petersen", "Adrian Cosma", "Jesus Colino", "Colin White", "Jacob Votava", "Vladimir Vinnikov", "Ethan Delaney", "Petr Spelda", "Vit Stritecky", "Syed M. Shahid", "Jean-Christophe Mourrat", "Lavr Vetoshkin", "Koen Sponselee", "Renas Bacho", "Zheng-Xin Yong", "Florencia de la Rosa", "Nathan Cho", "Xiuyu Li", "Guillaume Malod", "Orion Weller", "Guglielmo Albani", "Leon Lang", "Julien Laurendeau", "Dmitry Kazakov", "Fatimah Adesanya", "Julien Portier", "Lawrence Hollom", "Victor Souza", "Yuchen Anna Zhou", "Julien Degorre", "Yiğit Yalın", "Gbenga Daniel Obikoya", "Rai", "Filippo Bigi", "M. C. Boscá", "Oleg Shumar", "Kaniuar Bacho", "Gabriel Recchia", "Mara Popescu", "Nikita Shulga", "Ngefor Mildred Tanwie", "Thomas C. H. Lux", "Ben Rank", "Colin Ni", "Matthew Brooks", "Alesia Yakimchyk", "Huanxu", "Liu", "Stefano Cavalleri", "Olle Häggström", "Emil Verkama", "Joshua Newbould", "Hans Gundlach", "Leonor Brito-Santana", "Brian Amaro", "Vivek Vajipey", "Rynaa Grover", "Ting Wang", "Yosi Kratish", "Wen-Ding Li", "Sivakanth Gopi", "Andrea Caciolai", "Christian Schroeder de Witt", "Pablo Hernández-Cámara", "Emanuele Rodolà", "Jules Robins", "Dominic Williamson", "Vincent Cheng", "Brad Raynor", "Hao Qi", "Ben Segev", "Jingxuan Fan", "Sarah Martinson", "Erik Y. Wang", "Kaylie Hausknecht", "Michael P. Brenner", "Mao Mao", "Christoph Demian", "Peyman Kassani", "Xinyu Zhang", "David Avagian", "Eshawn Jessica Scipio", "Alon Ragoler", "Justin Tan", "Blake Sims", "Rebeka Plecnik", "Aaron Kirtland", "Omer Faruk Bodur", "D. P. Shinde", "Yan Carlos Leyva Labrador", "Zahra Adoul", "Mohamed Zekry", "Ali Karakoc", "Tania C. B. Santos", "Samir Shamseldeen", "Loukmane Karim", "Anna Liakhovitskaia", "Nate Resman", "Nicholas Farina", "Juan Carlos Gonzalez", "Gabe Maayan", "Earth Anderson", "Rodrigo De Oliveira Pena", "Elizabeth Kelley", "Hodjat Mariji", "Rasoul Pouriamanesh", "Wentao Wu", "Ross Finocchio", "Ismail Alarab", "Joshua Cole", "Danyelle Ferreira", "Bryan Johnson", "Mohammad Safdari", "Liangti Dai", "Siriphan Arthornthurasuk", "Isaac C. McAlister", "Alejandro José Moyano", "Alexey Pronin", "Jing Fan", "Angel Ramirez-Trinidad", "Yana Malysheva", "Daphiny Pottmaier", "Omid Taheri", "Stanley Stepanic", "Samuel Perry", "Luke Askew", "Raúl Adrián Huerta Rodríguez", "Ali M. R. Minissi", "Ricardo Lorena", "Krishnamurthy Iyer", "Arshad Anil Fasiludeen", "Ronald Clark", "Josh Ducey", "Matheus Piza", "Maja Somrak", "Eric Vergo", "Juehang Qin", "Benjámin Borbás", "Eric Chu", "Jack Lindsey", "Antoine Jallon", "I. M. J. McInnis", "Evan Chen", "Avi Semler", "Luk Gloor", "Tej Shah", "Marc Carauleanu", "Pascal Lauer", "Tran Đuc Huy", "Hossein Shahrtash", "Emilien Duc", "Lukas Lewark", "Assaf Brown", "Samuel Albanie", "Brian Weber", "Warren S. Vaz", "Pierre Clavier", "Yiyang Fan", "Gabriel Poesia Reis e Silva", "Long", "Lian", "Marcus Abramovitch", "Xi Jiang", "Sandra Mendoza", "Murat Islam", "Juan Gonzalez", "Vasilios Mavroudis", "Justin Xu", "Pawan Kumar", "Laxman Prasad Goswami", "Daniel Bugas", "Nasser Heydari", "Ferenc Jeanplong", "Thorben Jansen", "Antonella Pinto", "Archimedes Apronti", "Abdallah Galal", "Ng Ze-An", "Ankit Singh", "Tong Jiang", "Joan of Arc Xavier", "Kanu Priya Agarwal", "Mohammed Berkani", "Gang Zhang", "Zhehang Du", "Benedito Alves de Oliveira Junior", "Dmitry Malishev", "Nicolas Remy", "Taylor D. Hartman", "Tim Tarver", "Stephen Mensah", "Gautier Abou Loume", "Wiktor Morak", "Farzad Habibi", "Sarah Hoback", "Will Cai", "Javier Gimenez", "Roselynn Grace Montecillo", "Jakub Łucki", "Russell Campbell", "Asankhaya Sharma", "Khalida Meer", "Shreen Gul", "Daniel Espinosa Gonzalez", "Xavier Alapont", "Alex Hoover", "Gunjan Chhablani", "Freddie Vargus", "Arunim Agarwal", "Yibo Jiang", "Deepakkumar Patil", "David Outevsky", "Kevin Joseph Scaria", "Rajat Maheshwari", "Abdelkader Dendane", "Priti Shukla", "Ashley Cartwright", "Sergei Bogdanov", "Niels Mündler", "Sören Möller", "Luca Arnaboldi", "Kunvar Thaman", "Muhammad Rehan Siddiqi", "Prajvi Saxena", "Himanshu Gupta", "Tony Fruhauff", "Glen Sherman", "Mátyás Vincze", "Siranut Usawasutsakorn", "Dylan Ler", "Anil Radhakrishnan", "Innocent Enyekwe", "Sk Md Salauddin", "Jiang Muzhen", "Aleksandr Maksapetyan", "Vivien Rossbach", "Chris Harjadi", "Mohsen Bahaloohoreh", "Claire Sparrow", "Jasdeep Sidhu", "Sam Ali", "Song Bian", "John Lai", "Eric Singer", "Justine Leon Uro", "Greg Bateman", "Mohamed Sayed", "Ahmed Menshawy", "Darling Duclosel", "Dario Bezzi", "Yashaswini Jain", "Ashley Aaron", "Murat Tiryakioglu", "Sheeshram Siddh", "Keith Krenek", "Imad Ali Shah", "Jun Jin", "Scott Creighton", "Denis Peskoff", "Zienab EL-Wasif", "Ragavendran P", "Michael Richmond", "Joseph McGowan", "Tejal Patwardhan", "Hao-Yu Sun", "Ting Sun", "Nikola Zubić", "Samuele Sala", "Stephen Ebert", "Jean Kaddour", "Manuel Schottdorf", "Dianzhuo Wang", "Gerol Petruzella", "Alex Meiburg", "Tilen Medved", "Ali ElSheikh", "S Ashwin Hebbar", "Lorenzo Vaquero", "Xianjun Yang", "Jason Poulos", "Vilém Zouhar", "Sergey Bogdanik", "Mingfang Zhang", "Jorge Sanz-Ros", "David Anugraha", "Yinwei Dai", "Anh N. Nhu", "Xue Wang", "Ali Anil Demircali", "Zhibai Jia", "Yuyin Zhou", "Juncheng Wu", "Mike He", "Nitin Chandok", "Aarush Sinha", "Gaoxiang Luo", "Long Le", "Mickaël Noyé", "Michał Perełkiewicz", "Ioannis Pantidis", "Tianbo Qi", "Soham Sachin Purohit", "Letitia Parcalabescu", "Thai-Hoa Nguyen", "Genta Indra Winata", "Edoardo M. Ponti", "Hanchen Li", "Kaustubh Dhole", "Jongee Park", "Dario Abbondanza", "Yuanli Wang", "Anupam Nayak", "Diogo M. Caetano", "Antonio A. W. L. Wong", "Maria del Rio-Chanona", "Dániel Kondor", "Pieter Francois", "Ed Chalstrey", "Jakob Zsambok", "Dan Hoyer", "Jenny Reddish", "Jakob Hauser", "Francisco-Javier Rodrigo-Ginés", "Suchandra Datta", "Maxwell Shepherd", "Thom Kamphuis", "Qizheng Zhang", "Hyunjun Kim", "Ruiji Sun", "Jianzhu Yao", "Franck Dernoncourt", "Satyapriya Krishna", "Sina Rismanchian", "Bonan Pu", "Francesco Pinto", "Yingheng Wang", "Kumar Shridhar", "Kalon J. Overholt", "Glib Briia", "Hieu Nguyen", "David", "Soler Bartomeu", "Tony CY Pang", "Adam Wecker", "Yifan Xiong", "Fanfei Li", "Lukas S. Huber", "Joshua Jaeger", "Romano De Maddalena", "Xing Han Lù", "Yuhui Zhang", "Claas Beger", "Patrick Tser Jern Kon", "Sean Li", "Vivek Sanker", "Ming Yin", "Yihao Liang", "Xinlu Zhang", "Ankit Agrawal", "Li S. Yifei", "Zechen Zhang", "Mu Cai", "Yasin Sonmez", "Costin Cozianu", "Changhao Li", "Alex Slen", "Shoubin Yu", "Hyun Kyu Park", "Gabriele Sarti", "Marcin Briański", "Alessandro Stolfo", "Truong An Nguyen", "Mike Zhang", "Yotam Perlitz", "Jose Hernandez-Orallo", "Runjia Li", "Amin Shabani", "Felix Juefei-Xu", "Shikhar Dhingra", "Orr Zohar", "My Chiffon Nguyen", "Alexander Pondaven", "Abdurrahim Yilmaz", "Xuandong Zhao", "Chuanyang Jin", "Muyan Jiang", "Stefan Todoran", "Xinyao Han", "Jules Kreuer", "Brian Rabern", "Anna Plassart", "Martino Maggetti", "Luther Yap", "Robert Geirhos", "Jonathon Kean", "Dingsu Wang", "Sina Mollaei", "Chenkai Sun", "Yifan Yin", "Shiqi Wang", "Rui Li", "Yaowen Chang", "Anjiang Wei", "Alice Bizeul", "Xiaohan Wang", "Alexandre Oliveira Arrais", "Kushin Mukherjee", "Jorge Chamorro-Padial", "Jiachen Liu", "Xingyu Qu", "Junyi Guan", "Adam Bouyamourn", "Shuyu Wu", "Martyna Plomecka", "Junda Chen", "Mengze Tang", "Jiaqi Deng", "Shreyas Subramanian", "Haocheng Xi", "Haoxuan Chen", "Weizhi Zhang", "Yinuo Ren", "Haoqin Tu", "Sejong Kim", "Yushun Chen", "Sara Vera Marjanović", "Junwoo Ha", "Grzegorz Luczyna", "Jeff J. Ma", "Zewen Shen", "Dawn Song", "Cedegao E. Zhang", "Zhun Wang", "Gaël Gendron", "Yunze Xiao", "Leo Smucker", "Erica Weng", "Kwok Hao Lee", "Zhe Ye", "Stefano Ermon", "Ignacio D. Lopez-Miguel", "Theo Knights", "Anthony Gitter", "Namkyu Park", "Boyi Wei", "Hongzheng Chen", "Kunal Pai", "Ahmed Elkhanany", "Han Lin", "Philipp D. Siedler", "Jichao Fang", "Ritwik Mishra", "Károly Zsolnai-Fehér", "Xilin Jiang", "Shadab Khan", "Jun Yuan", "Rishab Kumar Jain", "Xi Lin", "Mike Peterson", "Zhe Wang", "Aditya Malusare", "Maosen Tang", "Isha Gupta", "Ivan Fosin", "Timothy Kang", "Barbara Dworakowska", "Kazuki Matsumoto", "Guangyao Zheng", "Gerben Sewuster", "Jorge Pretel Villanueva", "Ivan Rannev", "Igor Chernyavsky", "Jiale Chen", "Deepayan Banik", "Ben Racz", "Wenchao Dong", "Jianxin Wang", "Laila Bashmal", "Duarte V. Gonçalves", "Wei Hu", "Kaushik Bar", "Ondrej Bohdal", "Atharv Singh Patlan", "Shehzaad Dhuliawala", "Caroline Geirhos", "Julien Wist", "Yuval Kansal", "Bingsen Chen", "Kutay Tire", "Atak Talay Yücel", "Brandon Christof", "Veerupaksh Singla", "Zijian Song", "Sanxing Chen", "Jiaxin Ge", "Kaustubh Ponkshe", "Isaac Park", "Tianneng Shi", "Martin Q. Ma", "Joshua Mak", "Sherwin Lai", "Antoine Moulin", "Zhuo Cheng", "Zhanda Zhu", "Ziyi Zhang", "Vaidehi Patil", "Ketan Jha", "Qiutong Men", "Jiaxuan Wu", "Tianchi Zhang", "Bruno Hebling Vieira", "Alham Fikri Aji", "Jae-Won Chung", "Mohammed Mahfoud", "Ha Thi Hoang", "Marc Sperzel", "Wei Hao", "Kristof Meding", "Sihan Xu", "Vassilis Kostakos", "Davide Manini", "Yueying Liu", "Christopher Toukmaji", "Jay Paek", "Eunmi Yu", "Arif Engin Demircali", "Zhiyi Sun", "Ivan Dewerpe", "Hongsen Qin", "Roman Pflugfelder", "James Bailey", "Johnathan Morris", "Ville Heilala", "Sybille Rosset", "Zishun Yu", "Peter E. Chen", "Woongyeong Yeo", "Eeshaan Jain", "Ryan Yang", "Sreekar Chigurupati", "Julia Chernyavsky", "Sai Prajwal Reddy", "Subhashini Venugopalan", "Hunar Batra", "Core Francisco Park", "Hieu Tran", "Guilherme Maximiano", "Genghan Zhang", "Yizhuo Liang", "Hu Shiyu", "Rongwu Xu", "Rui Pan", "Siddharth Suresh", "Ziqi Liu", "Samaksh Gulati", "Songyang Zhang", "Peter Turchin", "Christopher W. Bartlett", "Christopher R. Scotese", "Phuong M. Cao", "Ben Wu", "Jacek Karwowski", "Davide Scaramuzza", "Aakaash Nattanmai", "Gordon McKellips", "Anish Cheraku", "Asim Suhail", "Ethan Luo", "Marvin Deng", "Jason Luo", "Ashley Zhang", "Kavin Jindel", "Jay Paek", "Kasper Halevy", "Allen Baranov", "Michael Liu", "Advaith Avadhanam", "David Zhang", "Vincent Cheng", "Brad Ma", "Evan Fu", "Liam Do", "Joshua Lass", "Hubert Yang", "Surya Sunkari", "Vishruth Bharath", "Violet Ai", "James Leung", "Rishit Agrawal", "Alan Zhou", "Kevin Chen", "Tejas Kalpathi", "Ziqi Xu", "Gavin Wang", "Tyler Xiao", "Erik Maung", "Sam Lee", "Ryan Yang", "Roy Yue", "Ben Zhao", "Julia Yoon", "Sunny Sun", "Aryan Singh", "Ethan Luo", "Clark Peng", "Tyler Osbey", "Taozhi Wang", "Daryl Echeazu", "Hubert Yang", "Timothy Wu", "Spandan Patel", "Vidhi Kulkarni", "Vijaykaarti Sundarapandiyan", "Ashley Zhang", "Andrew Le", "Zafir Nasim", "Srikar Yalam", "Ritesh Kasamsetty", "Soham Samal", "Hubert Yang", "David Sun", "Nihar Shah", "Abhijeet Saha", "Alex Zhang", "Leon Nguyen", "Laasya Nagumalli", "Kaixin Wang", "Alan Zhou", "Aidan Wu", "Jason Luo", "Anwith Telluri", "Steven Dillmann", "Zhengxiang Wang", "Junyu Luo", "Hugo Lunn", "Artem Gazizov", "Haitz Sáez de Ocáriz Borde", "Ivan Trus", "Morgan Hervault", "Zheyu Zhang", "Bo Chen", "Yuchen Wu", "Christopher J. Cordier", "Gün Kaynar", "Cansin Ayvaz", "Polina Avdiunina", "Johannes Brust", "Xingjian Diao", "K. D. Meaney", "Yifan Gu", "Chenyu Wang", "Chenzhuo Dong", "William Wright", "Simon Brave", "Owen Root", "Jiayuan Liu", "Chow Chun Lok", "Tianqin Li", "Shiyi Du", "Dailan He", "Lufeiya Liu", "Sina Jamalzadegan", "Anil Ramakrishna", "Xuanqing Xu", "Xin Qing", "Xin Luo", "Wenkai Li", "Shi Bo", "Filipp Gusev", "Maximos Skandalis", "Desheng Ma", "Chunhui Zhang", "Haoran Qiu", "Allen G Hart", "Rickard Brüel Gabrielsson", "Ido Akov", "Artem Lukoianov", "Summer Yue", "Alexandr Wang", "Dan Hendrycks"]
year: 2025
venue: "arXiv:2501.14249"
source: "https://arxiv.org/abs/2501.14249"
doi: "10.1038/s41586-025-09962-4"
arxiv: "2501.14249"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/phan-2025-humanity-s-last-exam/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/) / Humanity's Last Exam

# Humanity's Last Exam

**Authors:** [Long Phan](https://md.research.elamur.ai/free/en/authors/long-phan/), Alice Gatti, [Ziwen Han](https://md.research.elamur.ai/free/en/authors/ziwen-han/), [Nathaniel Li](https://md.research.elamur.ai/free/en/authors/nathaniel-li/), Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal et al. (1158 authors) · **Year:** 2025 · **Venue:** arXiv:2501.14249 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://doi.org/10.1038/s41586-025-09962-4) · [PDF](https://arxiv.org/pdf/2501.14249) · [DOI 10.1038/s41586-025-09962-4](https://doi.org/10.1038/s41586-025-09962-4) · [arXiv 2501.14249](https://arxiv.org/abs/2501.14249) · [OpenAlex](https://openalex.org/W4406840771)

**Citations:** 49 citations (openalex-snapshot · 2026-10-14) · [186](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Popular benchmarks such as MMLU are now close to saturated, so they no longer tell us much about how capable frontier LLMs have become. To fix this, the authors built Humanity's Last Exam (HLE), a multimodal benchmark of very hard, closed-ended academic questions. Each question has a single verifiable answer. Close to a thousand subject experts from many countries and institutions wrote questions across more than a hundred disciplines, with mathematics making up the largest share. Before a question was accepted, it was tested against leading models and dropped if they could solve it. Surviving questions then went through two rounds of expert review: the first improved submissions, and the second approved the best ones for the final set. Evaluation used a standard reasoning-then-answer prompt, with o3-mini judging whether answers were correct. Every model tested scored low and was badly calibrated, often giving wrong answers with high confidence. HLE therefore gives researchers and policymakers a shared, hard-to-game measure of expert-level knowledge and reasoning. The authors note that it does not measure open-ended research ability.

## Key points
- The final public set has about 2,500 questions, a mix of exact-match and multiple-choice. Some include images. A private held-out portion is kept to detect overfitting.
- Accuracy (%) on HLE: GPT-4o 2.7, Grok 2 3.0, Claude 3.5 Sonnet 4.1, Gemini 1.5 Pro 4.6, Gemini 2.0 Flash Thinking 6.6, o1 8.0. DeepSeek-R1 scored 8.5 and o3-mini (high) 13.4, both on the text-only subset.
- RMS calibration error ranged from about 73% to 89%. This means the models' stated confidence was far above their real accuracy, a sign of hallucination.
- Reasoning models produced many more completion tokens than non-reasoning models for their modest gains. The authors call for compute-efficient progress, not only higher accuracy.
- The authors thought scores above 50% by the end of 2025 were plausible. Even so, they argue that a high HLE score would not show AGI or autonomous research ability.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Mathematics & statistics](https://md.research.elamur.ai/free/en/shelves/mathematics/)

## Cited by

Cited by

186 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 175 not yet.

- [gpt-oss-120b & gpt-oss-20b Model Card](https://md.research.elamur.ai/free/en/papers/agarwal-2025-gpt-oss-120b-gpt-oss/)
- [Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2025-towards-reasoning-era-survey-long/)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://md.research.elamur.ai/free/en/papers/comanici-2025-gemini-2-5-pushing-frontier/)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://md.research.elamur.ai/free/en/papers/deepseek-2025-ai-deepseek-v3-2-pushing/)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-2026-ai-deepseek-v4-highly-efficient/)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://md.research.elamur.ai/free/en/papers/glm-2025-4-glm-4-5-agentic/)
- [OpenThoughts: Data Recipes for Reasoning Models](https://md.research.elamur.ai/free/en/papers/guha-2025-openthoughts-data-recipes-reasoning-models/)
- [Kimi K2: Open Agentic Intelligence](https://md.research.elamur.ai/free/en/papers/kimi-2025-team-kimi-k2-open-agentic/)
- [s1: Simple test-time scaling](https://md.research.elamur.ai/free/en/papers/muennighoff-2025-s1-simple-test-time-scaling/)
- [HistoriQA-ThirdRepublic: Multi-Hop Question Answering Corpus for Historical Research, Parliamentary Debates from the French Third Republic (1870-1940)](https://md.research.elamur.ai/free/en/papers/pellet-2026-historiqa-thirdrepublic-multi-hop-question/)
- [Large Language Models for Information Retrieval: A Survey](https://md.research.elamur.ai/free/en/papers/zhu-2023-large-language-models-information-retrieval/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{phan2025humanity,
  title = {{Humanity's Last Exam}},
  author = {Phan, Long and Gatti, Alice and Han, Ziwen and Li, Nathaniel and Hu, Josephina and Zhang, Hugh and Zhang, Chen Bo Calvin and Shaaban, Mohamed and Ling, John and Shi, Sean and Choi, Michael and Agrawal, Anish and Chopra, Arnav and Khoja, Adam and Kim, Ryan and Ren, Richard and Hausenloy, Jason and Zhang, Oliver and Mazeika, Mantas and Dodonov, Dmitry and Nguyen, Tung and Lee, Jaeho and Anderson, Daron and Doroshenko, Mikhail and Stokes, Alun Cennyth and Mahmood, Mobeen and Pokutnyi, Oleksandr and Iskra, Oleg and Wang, Jessica P. and Levin, John-Clark and Kazakov, Mstyslav and Feng, Fiona and Feng, Steven Y. and Zhao, Haoran and Yu, Michael and Gangal, Varun and Zou, Chelsea and Wang, Zihan and Popov, Serguei and Gerbicz, Robert and Galgon, Geoff and Schmitt, Johannes and Yeadon, Will and Lee, Yongki and Sauers, Scott and Sanchez, Alvaro and Giska, Fabian and Roth, Marc and Riis, Søren and Utpala, Saiteja and Burns, Noah and Goshu, Gashaw M. and Naiya, Mohinder Maheshbhai and Agu, Chidozie and Giboney, Zachary and Cheatom, Antrell and Fournier-Facio, Francesco and Crowson, Sarah-Jane and Finke, Lennart and Cheng, Zerui and Zampese, Jennifer and Hoerr, Ryan G. and Nandor, Mark and Park, Hyunwoo and Gehrunger, Tim and Cai, Jiaqi and McCarty, Ben and Garretson, Alexis C and Taylor, Edwin and Sileo, Damien and Ren, Qiuyu and Qazi, Usman and Li, Lianghui and Nam, Jungbae and Wydallis, John B. and Arkhipov, Pavel and Shi, Jack Wei Lun and Bacho, Aras and Willcocks, Chris G. and Cao, Hangrui and Motwani, Sumeet and Santos, Emily de Oliveira and Veith, Johannes and Vendrow, Edward and Cojoc, Doru and Zenitani, Kengo and Robinson, Joshua and Tang, Longke and Li, Yuqi and Vendrow, Joshua and Fraga, Natanael Wildner and Kuchkin, Vladyslav and Maksimov, Andrey Pupasov and Marion, Pierre and Efremov, Denis and Lynch, Jayson and Liang, Kaiqu and Mikov, Aleksandar and Gritsevskiy, Andrew and Guillod, Julien and Demir, Gözdenur and Martinez, Dakotah and Pageler, Ben and Zhou, Kevin and Soori, Saeed and Press, Ori and Tang, Henry and Rissone, Paolo and Green, Sean R. and Brüssel, Lina and Twayana, Moon and Dieuleveut, Aymeric and Imperial, Joseph Marvin and Prabhu, Ameya and Yang, Jinzhou and Crispino, Nick and Rao, Arun and Zvonkine, Dimitri and Loiseau, Gabriel and Kalinin, Mikhail and Lukas, Marco and Manolescu, Ciprian and Stambaugh, Nate and Mishra, Subrata and Hogg, Tad and Bosio, Carlo and Coppola, Brian P and Salazar, Julian and Jin, Jaehyeok and Sayous, Rafael and Ivanov, Stefan and Schwaller, Philippe and Senthilkuma, Shaipranesh and Bran, Andres M and Algaba, Andres and Houte, Kelsey Van den and Sypt, Lynn Van Der and Verbeken, Brecht and Noever, David and Kopylov, Alexei and Myklebust, Benjamin and Li, Bikun and Schut, Lisa and Zheltonozhskii, Evgenii and Yuan, Qiaochu and Lim, Derek and Stanley, Richard and Yang, Tong and Maar, John and Wykowski, Julian and Oller, Martí and Sahu, Anmol and Ardito, Cesare Giulio and Hu, Yuzheng and Kamdoum, Ariel Ghislain Kemogne and Jin, Alvin and Vilchis, Tobias Garcia and Zu, Yuexuan and Lackner, Martin and Koppel, James and Sun, Gongbo and Antonenko, Daniil S. and Chern, Steffi and Zhao, Bingchen and Arsene, Pierrot and Cavanagh, Joseph M and Li, Daofeng and Shen, Jiawei and Crisostomi, Donato and Zhang, Wenjin and Dehghan, Ali and Ivanov, Sergey and Perrella, David and Kaparov, Nurdin and Zang, Allen and Sucholutsky, Ilia and Kharlamova, Arina and Orel, Daniil and Poritski, Vladislav and Ben-David, Shalev and Berger, Zachary and Whitfill, Parker and Foster, Michael and Munro, Daniel and Ho, Linh and Sivarajan, Shankar and Hava, Dan Bar and Kuchkin, Aleksey and Holmes, David and Rodriguez-Romero, Alexandra and Sommerhage, Frank and Zhang, Anji and Moat, Richard and Schneider, Keith and Kazibwe, Zakayo and Clarke, Don and Kim, Dae Hyun and Dias, Felipe Meneguitti and Fish, Sara and Elser, Veit and Kreiman, Tobias and Vilchis, Victor Efren Guadarrama and Klose, Immo and Anantheswaran, Ujjwala and Zweiger, Adam and Rawal, Kaivalya and Li, Jeffery and Nguyen, Jeremy and Daans, Nicolas and Heidinger, Haline and Radionov, Maksim and Rozhoň, Václav and Ginis, Vincent and Stump, Christian and Cohen, Niv and Poświata, Rafał and Tkadlec, Josef and Goldfarb, Alan and Wang, Chenguang and Padlewski, Piotr and Barzowski, Stanislaw and Montgomery, Kyle and Stendall, Ryan and Tucker-Foltz, Jamie and Stade, Jack and Rogers, T. Ryan and Goertzen, Tom and Grabb, Declan and Shukla, Abhishek and Givré, Alan and Ambay, John Arnold and Sen, Archan and Aziz, Muhammad Fayez and Inlow, Mark H and He, Hao and Zhang, Ling and Kaddar, Younesse and Ängquist, Ivar and Chen, Yanxu and Wang, Harrison K and Ramakrishnan, Kalyan and Thornley, Elliott and Terpin, Antonio and Schoelkopf, Hailey and Zheng, Eric and Carmi, Avishy and Brown, Ethan D. L. and Zhu, Kelin and Bartolo, Max and Wheeler, Richard and Stehberger, Martin and Bradshaw, Peter and Heimonen, JP and Sridhar, Kaustubh and Akov, Ido and Sandlin, Jennifer and Makarychev, Yury and Tam, Joanna and Hoang, Hieu and Cunningham, David M. and Goryachev, Vladimir and Patramanis, Demosthenes and Krause, Michael and Redenti, Andrew and Aldous, David and Lai, Jesyin and Coleman, Shannon and Xu, Jiangnan and Lee, Sangwon and Magoulas, Ilias and Zhao, Sandy and Tang, Ning and Cohen, Michael K. and Paradise, Orr and Kirchner, Jan Hendrik and Ovchynnikov, Maksym and Matos, Jason O. and Shenoy, Adithya and Wang, Michael and Nie, Yuzhou and Sztyber-Betley, Anna and Faraboschi, Paolo and Riblet, Robin and Crozier, Jonathan and Halasyamani, Shiv and Verma, Shreyas and Joshi, Prashant and Meril, Eli and Ma, Ziqiao and Andréoletti, Jérémy and Singhal, Raghav and Platnick, Jacob and Nevirkovets, Volodymyr and Basler, Luke and Ivanov, Alexander and Khoury, Seri and Gustafsson, Nils and Piccardo, Marco and Mostaghimi, Hamid and Chen, Qijia and Singh, Virendra and Khánh, Tran Quoc and Rosu, Paul and Szlyk, Hannah and Brown, Zachary and Narayan, Himanshu and Menezes, Aline and Roberts, Jonathan and Alley, William and Sun, Kunyang and Patel, Arkil and Lamparth, Max and Reuel, Anka and Xin, Linwei and Xu, Hanmeng and Loader, Jacob and Martin, Freddie and Wang, Zixuan and Achilleos, Andrea and Preu, Thomas and Korbak, Tomek and Bosio, Ida and Kazemi, Fereshteh and Chen, Ziye and Bálint, Biró and Lo, Eve J. Y. and Wang, Jiaqi and Nunes, Maria Inês S. and Milbauer, Jeremiah and Bari, M Saiful and Wang, Zihao and Ansarinejad, Behzad and Sun, Yewen and Durand, Stephane and Elgnainy, Hossam and Douville, Guillaume and Tordera, Daniel and Balabanian, George and Wolff, Hew and Kvistad, Lynna and Milliron, Hsiaoyun and Sakor, Ahmad and Eron, Murat and O and Shah, Shailesh and Zhou, Xiaoxiang and Kamalov, Firuz and Abdoli, Sherwin and Santens, Tim and Barkan, Shaul and Tee, Allison and Zhang, Robin and Tomasiello, Alessandro and Luca, G. Bruno De and Looi, Shi-Zhuo and Le, Vinh-Kha and Kolt, Noam and Pan, Jiayi and Rodman, Emma and Drori, Jacob and Fossum, Carl J and Muennighoff, Niklas and Jagota, Milind and Pradeep, Ronak and Fan, Honglu and Eicher, Jonathan and Chen, Michael and Thaman, Kushal and Merrill, William and Firsching, Moritz and Harris, Carter and Ciobâcă, Stefan and Gross, Jason and Pandey, Rohan and Gusev, Ilya and Jones, Adam and Agnihotri, Shashank and Zhelnov, Pavel and Mofayezi, Mohammadreza and Piperski, Alexander and Zhang, David K. and Dobarskyi, Kostiantyn and Leventov, Roman and Soroko, Ignat and Duersch, Joshua and Taamazyan, Vage and Ho, Andrew and Ma, Wenjie and Held, William and Xian, Ruicheng and Zebaze, Armel Randy and Mohamed, Mohanad and Leser, Julian Noah and Yuan, Michelle X and Yacar, Laila and Lengler, Johannes and Olszewska, Katarzyna and Fratta, Claudio Di and Oliveira, Edson and Jackson, Joseph W. and Zou, Andy and Chidambaram, Muthu and Manik, Timothy and Haffenden, Hector and Stander, Dashiell and Dasouqi, Ali and Shen, Alexander and Golshani, Bita and Stap, David and Kretov, Egor and Uzhou, Mikalai and Zhidkovskaya, Alina Borisovna and Winter, Nick and Rodriguez, Miguel Orbegozo and Lauff, Robert and Wehr, Dustin and Tang, Colin and Hossain, Zaki and Phillips, Shaun and Samuele, Fortuna and Ekström, Fredrik and Hammon, Angela and Patel, Oam and Farhidi, Faraz and Medley, George and Mohammadzadeh, Forough and Peñaflor, Madellene and Kassahun, Haile and Friedrich, Alena and Perez, Rayner Hernandez and Pyda, Daniel and Sakal, Taom and Dhamane, Omkar and Mirabadi, Ali Khajegili and Hallman, Eric and Okutsu, Kenchi and Battaglia, Mike and Maghsoudimehrabani, Mohammad and Amit, Alon and Hulbert, Dave and Pereira, Roberto and Weber, Simon and Handoko and Peristyy, Anton and Malina, Stephen and Mehkary, Mustafa and Aly, Rami and Reidegeld, Frank and Dick, Anna-Katharina and Friday, Cary and Singh, Mukhwinder and Shapourian, Hassan and Kim, Wanyoung and Costa, Mariana and Gurdogan, Hubeyb and Kumar, Harsh and Ceconello, Chiara and Zhuang, Chao and Park, Haon and Carroll, Micah and Tawfeek, Andrew R. and Steinerberger, Stefan and Aggarwal, Daattavya and Kirchhof, Michael and Dai, Linjie and Kim, Evan and Ferret, Johan and Shah, Jainam and Wang, Yuzhou and Yan, Minghao and Burdzy, Krzysztof and Zhang, Lixin and Franca, Antonio and Pham, Diana T. and Loh, Kang Yong and Robinson, Joshua and Jackson, Abram and Giordano, Paolo and Petersen, Philipp and Cosma, Adrian and Colino, Jesus and White, Colin and Votava, Jacob and Vinnikov, Vladimir and Delaney, Ethan and Spelda, Petr and Stritecky, Vit and Shahid, Syed M. and Mourrat, Jean-Christophe and Vetoshkin, Lavr and Sponselee, Koen and Bacho, Renas and Yong, Zheng-Xin and Rosa, Florencia de la and Cho, Nathan and Li, Xiuyu and Malod, Guillaume and Weller, Orion and Albani, Guglielmo and Lang, Leon and Laurendeau, Julien and Kazakov, Dmitry and Adesanya, Fatimah and Portier, Julien and Hollom, Lawrence and Souza, Victor and Zhou, Yuchen Anna and Degorre, Julien and Yalın, Yiğit and Obikoya, Gbenga Daniel and Rai and Bigi, Filippo and Boscá, M. C. and Shumar, Oleg and Bacho, Kaniuar and Recchia, Gabriel and Popescu, Mara and Shulga, Nikita and Tanwie, Ngefor Mildred and Lux, Thomas C. H. and Rank, Ben and Ni, Colin and Brooks, Matthew and Yakimchyk, Alesia and Huanxu and Liu and Cavalleri, Stefano and Häggström, Olle and Verkama, Emil and Newbould, Joshua and Gundlach, Hans and Brito-Santana, Leonor and Amaro, Brian and Vajipey, Vivek and Grover, Rynaa and Wang, Ting and Kratish, Yosi and Li, Wen-Ding and Gopi, Sivakanth and Caciolai, Andrea and Witt, Christian Schroeder de and Hernández-Cámara, Pablo and Rodolà, Emanuele and Robins, Jules and Williamson, Dominic and Cheng, Vincent and Raynor, Brad and Qi, Hao and Segev, Ben and Fan, Jingxuan and Martinson, Sarah and Wang, Erik Y. and Hausknecht, Kaylie and Brenner, Michael P. and Mao, Mao and Demian, Christoph and Kassani, Peyman and Zhang, Xinyu and Avagian, David and Scipio, Eshawn Jessica and Ragoler, Alon and Tan, Justin and Sims, Blake and Plecnik, Rebeka and Kirtland, Aaron and Bodur, Omer Faruk and Shinde, D. P. and Labrador, Yan Carlos Leyva and Adoul, Zahra and Zekry, Mohamed and Karakoc, Ali and Santos, Tania C. B. and Shamseldeen, Samir and Karim, Loukmane and Liakhovitskaia, Anna and Resman, Nate and Farina, Nicholas and Gonzalez, Juan Carlos and Maayan, Gabe and Anderson, Earth and Pena, Rodrigo De Oliveira and Kelley, Elizabeth and Mariji, Hodjat and Pouriamanesh, Rasoul and Wu, Wentao and Finocchio, Ross and Alarab, Ismail and Cole, Joshua and Ferreira, Danyelle and Johnson, Bryan and Safdari, Mohammad and Dai, Liangti and Arthornthurasuk, Siriphan and McAlister, Isaac C. and Moyano, Alejandro José and Pronin, Alexey and Fan, Jing and Ramirez-Trinidad, Angel and Malysheva, Yana and Pottmaier, Daphiny and Taheri, Omid and Stepanic, Stanley and Perry, Samuel and Askew, Luke and Rodríguez, Raúl Adrián Huerta and Minissi, Ali M. R. and Lorena, Ricardo and Iyer, Krishnamurthy and Fasiludeen, Arshad Anil and Clark, Ronald and Ducey, Josh and Piza, Matheus and Somrak, Maja and Vergo, Eric and Qin, Juehang and Borbás, Benjámin and Chu, Eric and Lindsey, Jack and Jallon, Antoine and McInnis, I. M. J. and Chen, Evan and Semler, Avi and Gloor, Luk and Shah, Tej and Carauleanu, Marc and Lauer, Pascal and Huy, Tran Đuc and Shahrtash, Hossein and Duc, Emilien and Lewark, Lukas and Brown, Assaf and Albanie, Samuel and Weber, Brian and Vaz, Warren S. and Clavier, Pierre and Fan, Yiyang and Silva, Gabriel Poesia Reis e and Long and Lian and Abramovitch, Marcus and Jiang, Xi and Mendoza, Sandra and Islam, Murat and Gonzalez, Juan and Mavroudis, Vasilios and Xu, Justin and Kumar, Pawan and Goswami, Laxman Prasad and Bugas, Daniel and Heydari, Nasser and Jeanplong, Ferenc and Jansen, Thorben and Pinto, Antonella and Apronti, Archimedes and Galal, Abdallah and Ze-An, Ng and Singh, Ankit and Jiang, Tong and Xavier, Joan of Arc and Agarwal, Kanu Priya and Berkani, Mohammed and Zhang, Gang and Du, Zhehang and Junior, Benedito Alves de Oliveira and Malishev, Dmitry and Remy, Nicolas and Hartman, Taylor D. and Tarver, Tim and Mensah, Stephen and Loume, Gautier Abou and Morak, Wiktor and Habibi, Farzad and Hoback, Sarah and Cai, Will and Gimenez, Javier and Montecillo, Roselynn Grace and Łucki, Jakub and Campbell, Russell and Sharma, Asankhaya and Meer, Khalida and Gul, Shreen and Gonzalez, Daniel Espinosa and Alapont, Xavier and Hoover, Alex and Chhablani, Gunjan and Vargus, Freddie and Agarwal, Arunim and Jiang, Yibo and Patil, Deepakkumar and Outevsky, David and Scaria, Kevin Joseph and Maheshwari, Rajat and Dendane, Abdelkader and Shukla, Priti and Cartwright, Ashley and Bogdanov, Sergei and Mündler, Niels and Möller, Sören and Arnaboldi, Luca and Thaman, Kunvar and Siddiqi, Muhammad Rehan and Saxena, Prajvi and Gupta, Himanshu and Fruhauff, Tony and Sherman, Glen and Vincze, Mátyás and Usawasutsakorn, Siranut and Ler, Dylan and Radhakrishnan, Anil and Enyekwe, Innocent and Salauddin, Sk Md and Muzhen, Jiang and Maksapetyan, Aleksandr and Rossbach, Vivien and Harjadi, Chris and Bahaloohoreh, Mohsen and Sparrow, Claire and Sidhu, Jasdeep and Ali, Sam and Bian, Song and Lai, John and Singer, Eric and Uro, Justine Leon and Bateman, Greg and Sayed, Mohamed and Menshawy, Ahmed and Duclosel, Darling and Bezzi, Dario and Jain, Yashaswini and Aaron, Ashley and Tiryakioglu, Murat and Siddh, Sheeshram and Krenek, Keith and Shah, Imad Ali and Jin, Jun and Creighton, Scott and Peskoff, Denis and EL-Wasif, Zienab and P, Ragavendran and Richmond, Michael and McGowan, Joseph and Patwardhan, Tejal and Sun, Hao-Yu and Sun, Ting and Zubić, Nikola and Sala, Samuele and Ebert, Stephen and Kaddour, Jean and Schottdorf, Manuel and Wang, Dianzhuo and Petruzella, Gerol and Meiburg, Alex and Medved, Tilen and ElSheikh, Ali and Hebbar, S Ashwin and Vaquero, Lorenzo and Yang, Xianjun and Poulos, Jason and Zouhar, Vilém and Bogdanik, Sergey and Zhang, Mingfang and Sanz-Ros, Jorge and Anugraha, David and Dai, Yinwei and Nhu, Anh N. and Wang, Xue and Demircali, Ali Anil and Jia, Zhibai and Zhou, Yuyin and Wu, Juncheng and He, Mike and Chandok, Nitin and Sinha, Aarush and Luo, Gaoxiang and Le, Long and Noyé, Mickaël and Perełkiewicz, Michał and Pantidis, Ioannis and Qi, Tianbo and Purohit, Soham Sachin and Parcalabescu, Letitia and Nguyen, Thai-Hoa and Winata, Genta Indra and Ponti, Edoardo M. and Li, Hanchen and Dhole, Kaustubh and Park, Jongee and Abbondanza, Dario and Wang, Yuanli and Nayak, Anupam and Caetano, Diogo M. and Wong, Antonio A. W. L. and Rio-Chanona, Maria del and Kondor, Dániel and Francois, Pieter and Chalstrey, Ed and Zsambok, Jakob and Hoyer, Dan and Reddish, Jenny and Hauser, Jakob and Rodrigo-Ginés, Francisco-Javier and Datta, Suchandra and Shepherd, Maxwell and Kamphuis, Thom and Zhang, Qizheng and Kim, Hyunjun and Sun, Ruiji and Yao, Jianzhu and Dernoncourt, Franck and Krishna, Satyapriya and Rismanchian, Sina and Pu, Bonan and Pinto, Francesco and Wang, Yingheng and Shridhar, Kumar and Overholt, Kalon J. and Briia, Glib and Nguyen, Hieu and David and Bartomeu, Soler and Pang, Tony CY and Wecker, Adam and Xiong, Yifan and Li, Fanfei and Huber, Lukas S. and Jaeger, Joshua and Maddalena, Romano De and Lù, Xing Han and Zhang, Yuhui and Beger, Claas and Kon, Patrick Tser Jern and Li, Sean and Sanker, Vivek and Yin, Ming and Liang, Yihao and Zhang, Xinlu and Agrawal, Ankit and Yifei, Li S. and Zhang, Zechen and Cai, Mu and Sonmez, Yasin and Cozianu, Costin and Li, Changhao and Slen, Alex and Yu, Shoubin and Park, Hyun Kyu and Sarti, Gabriele and Briański, Marcin and Stolfo, Alessandro and Nguyen, Truong An and Zhang, Mike and Perlitz, Yotam and Hernandez-Orallo, Jose and Li, Runjia and Shabani, Amin and Juefei-Xu, Felix and Dhingra, Shikhar and Zohar, Orr and Nguyen, My Chiffon and Pondaven, Alexander and Yilmaz, Abdurrahim and Zhao, Xuandong and Jin, Chuanyang and Jiang, Muyan and Todoran, Stefan and Han, Xinyao and Kreuer, Jules and Rabern, Brian and Plassart, Anna and Maggetti, Martino and Yap, Luther and Geirhos, Robert and Kean, Jonathon and Wang, Dingsu and Mollaei, Sina and Sun, Chenkai and Yin, Yifan and Wang, Shiqi and Li, Rui and Chang, Yaowen and Wei, Anjiang and Bizeul, Alice and Wang, Xiaohan and Arrais, Alexandre Oliveira and Mukherjee, Kushin and Chamorro-Padial, Jorge and Liu, Jiachen and Qu, Xingyu and Guan, Junyi and Bouyamourn, Adam and Wu, Shuyu and Plomecka, Martyna and Chen, Junda and Tang, Mengze and Deng, Jiaqi and Subramanian, Shreyas and Xi, Haocheng and Chen, Haoxuan and Zhang, Weizhi and Ren, Yinuo and Tu, Haoqin and Kim, Sejong and Chen, Yushun and Marjanović, Sara Vera and Ha, Junwoo and Luczyna, Grzegorz and Ma, Jeff J. and Shen, Zewen and Song, Dawn and Zhang, Cedegao E. and Wang, Zhun and Gendron, Gaël and Xiao, Yunze and Smucker, Leo and Weng, Erica and Lee, Kwok Hao and Ye, Zhe and Ermon, Stefano and Lopez-Miguel, Ignacio D. and Knights, Theo and Gitter, Anthony and Park, Namkyu and Wei, Boyi and Chen, Hongzheng and Pai, Kunal and Elkhanany, Ahmed and Lin, Han and Siedler, Philipp D. and Fang, Jichao and Mishra, Ritwik and Zsolnai-Fehér, Károly and Jiang, Xilin and Khan, Shadab and Yuan, Jun and Jain, Rishab Kumar and Lin, Xi and Peterson, Mike and Wang, Zhe and Malusare, Aditya and Tang, Maosen and Gupta, Isha and Fosin, Ivan and Kang, Timothy and Dworakowska, Barbara and Matsumoto, Kazuki and Zheng, Guangyao and Sewuster, Gerben and Villanueva, Jorge Pretel and Rannev, Ivan and Chernyavsky, Igor and Chen, Jiale and Banik, Deepayan and Racz, Ben and Dong, Wenchao and Wang, Jianxin and Bashmal, Laila and Gonçalves, Duarte V. and Hu, Wei and Bar, Kaushik and Bohdal, Ondrej and Patlan, Atharv Singh and Dhuliawala, Shehzaad and Geirhos, Caroline and Wist, Julien and Kansal, Yuval and Chen, Bingsen and Tire, Kutay and Yücel, Atak Talay and Christof, Brandon and Singla, Veerupaksh and Song, Zijian and Chen, Sanxing and Ge, Jiaxin and Ponkshe, Kaustubh and Park, Isaac and Shi, Tianneng and Ma, Martin Q. and Mak, Joshua and Lai, Sherwin and Moulin, Antoine and Cheng, Zhuo and Zhu, Zhanda and Zhang, Ziyi and Patil, Vaidehi and Jha, Ketan and Men, Qiutong and Wu, Jiaxuan and Zhang, Tianchi and Vieira, Bruno Hebling and Aji, Alham Fikri and Chung, Jae-Won and Mahfoud, Mohammed and Hoang, Ha Thi and Sperzel, Marc and Hao, Wei and Meding, Kristof and Xu, Sihan and Kostakos, Vassilis and Manini, Davide and Liu, Yueying and Toukmaji, Christopher and Paek, Jay and Yu, Eunmi and Demircali, Arif Engin and Sun, Zhiyi and Dewerpe, Ivan and Qin, Hongsen and Pflugfelder, Roman and Bailey, James and Morris, Johnathan and Heilala, Ville and Rosset, Sybille and Yu, Zishun and Chen, Peter E. and Yeo, Woongyeong and Jain, Eeshaan and Yang, Ryan and Chigurupati, Sreekar and Chernyavsky, Julia and Reddy, Sai Prajwal and Venugopalan, Subhashini and Batra, Hunar and Park, Core Francisco and Tran, Hieu and Maximiano, Guilherme and Zhang, Genghan and Liang, Yizhuo and Shiyu, Hu and Xu, Rongwu and Pan, Rui and Suresh, Siddharth and Liu, Ziqi and Gulati, Samaksh and Zhang, Songyang and Turchin, Peter and Bartlett, Christopher W. and Scotese, Christopher R. and Cao, Phuong M. and Wu, Ben and Karwowski, Jacek and Scaramuzza, Davide and Nattanmai, Aakaash and McKellips, Gordon and Cheraku, Anish and Suhail, Asim and Luo, Ethan and Deng, Marvin and Luo, Jason and Zhang, Ashley and Jindel, Kavin and Paek, Jay and Halevy, Kasper and Baranov, Allen and Liu, Michael and Avadhanam, Advaith and Zhang, David and Cheng, Vincent and Ma, Brad and Fu, Evan and Do, Liam and Lass, Joshua and Yang, Hubert and Sunkari, Surya and Bharath, Vishruth and Ai, Violet and Leung, James and Agrawal, Rishit and Zhou, Alan and Chen, Kevin and Kalpathi, Tejas and Xu, Ziqi and Wang, Gavin and Xiao, Tyler and Maung, Erik and Lee, Sam and Yang, Ryan and Yue, Roy and Zhao, Ben and Yoon, Julia and Sun, Sunny and Singh, Aryan and Luo, Ethan and Peng, Clark and Osbey, Tyler and Wang, Taozhi and Echeazu, Daryl and Yang, Hubert and Wu, Timothy and Patel, Spandan and Kulkarni, Vidhi and Sundarapandiyan, Vijaykaarti and Zhang, Ashley and Le, Andrew and Nasim, Zafir and Yalam, Srikar and Kasamsetty, Ritesh and Samal, Soham and Yang, Hubert and Sun, David and Shah, Nihar and Saha, Abhijeet and Zhang, Alex and Nguyen, Leon and Nagumalli, Laasya and Wang, Kaixin and Zhou, Alan and Wu, Aidan and Luo, Jason and Telluri, Anwith and Dillmann, Steven and Wang, Zhengxiang and Luo, Junyu and Lunn, Hugo and Gazizov, Artem and Borde, Haitz Sáez de Ocáriz and Trus, Ivan and Hervault, Morgan and Zhang, Zheyu and Chen, Bo and Wu, Yuchen and Cordier, Christopher J. and Kaynar, Gün and Ayvaz, Cansin and Avdiunina, Polina and Brust, Johannes and Diao, Xingjian and Meaney, K. D. and Gu, Yifan and Wang, Chenyu and Dong, Chenzhuo and Wright, William and Brave, Simon and Root, Owen and Liu, Jiayuan and Lok, Chow Chun and Li, Tianqin and Du, Shiyi and He, Dailan and Liu, Lufeiya and Jamalzadegan, Sina and Ramakrishna, Anil and Xu, Xuanqing and Qing, Xin and Luo, Xin and Li, Wenkai and Bo, Shi and Gusev, Filipp and Skandalis, Maximos and Ma, Desheng and Zhang, Chunhui and Qiu, Haoran and Hart, Allen G and Gabrielsson, Rickard Brüel and Akov, Ido and Lukoianov, Artem and Yue, Summer and Wang, Alexandr and Hendrycks, Dan},
  year = {2025},
  doi = {10.1038/s41586-025-09962-4},
  eprint = {2501.14249},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1038/s41586-025-09962-4}
}
```

- [← previous How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmark](https://md.research.elamur.ai/free/en/papers/yang-2025-llm-reasoning-distracted-irrelevant-context/)
- [next → ICLE-RC: International Corpus of Learner English for Relative Clauses](https://md.research.elamur.ai/free/en/papers/das-2025-icle-rc-international-corpus-learner/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/)

## References

63 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. C. Alberti, K. Lee, and M. Collins. A bert baseline for the natural questions, 2019. URL https://arxiv.org/abs/1901.08634. — [arXiv:1901.08634](https://arxiv.org/abs/1901.08634) [↗](https://arxiv.org/abs/1901.08634)
2. M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y. Gal, and X. Davies. Agentharm: A benchmark for measuring harmfulness of llm agents, 2024. URL https://arxiv.org/abs/ 2410.09024. [↗](https://arxiv.org/abs/)
3. Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024. URL [https://api.semanticscholar.org/CorpusID:268232499](https://api.semanticscholar.org/CorpusID:268232499). [↗](https://api.semanticscholar.org/CorpusID:268232499)
4. Anthropic. Model card addendum: Claude 3.5 haiku and upgraded claude 3.5 sonnet, 2024. URL [https://assets.anthropic.com/m/1cd9d098ac3e6467/original/](https://assets.anthropic.com/m/1cd9d098ac3e6467/original/) Claude-3-Model-Card-October-Addendum.pdf. [↗](https://assets.anthropic.com/m/1cd9d098ac3e6467/original/)
5. Anthropic. Responsible scaling policy updates, 2024. URL [https://www.anthropic.com/](https://www.anthropic.com/) rsp-updates. [↗](https://www.anthropic.com/)
6. R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, J. Heidecke, and K. Singhal. Healthbench: Evaluating large language models towards improved human health, 2025. URL https://arxiv.org/abs/ 2505.08775. [↗](https://arxiv.org/abs/)
7. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021. URL https://arxiv.org/abs/2108.07732. [↗](https://arxiv.org/abs/2108.07732) [our card](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/)
8. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, N. Joseph, S. Kadavath, J. Kernion, T. Conerly, S. El-Showk, N. Elhage, Z. Hatfield-Dodds, D. Hernandez, T. Hume, S. Johnston, S. Kravec, L. Lovitt, N. Nanda, C. Olsson, D. Amodei, T. Brown, J.… [find](https://scholar.google.com/scholar?q=Y%2E+Bai%2C+A%2E+Jones%2C+K%2E+Ndousse%2C+A%2E+Askell%2C+A%2E+Chen%2C+N%2E+DasSarma%2C+D%2E+Drain%2C+S%2E+Fort%2C+D%2E+Ganguli%2C+T%2E+Henighan%2C+N%2E+Joseph%2C+S%2E+Kadavath%2C+J%2E+Kernion%2C+T%2E+Conerly%2C+S%2E+El%2DShowk%2C+N%2E+Elhage%2C+Z%2E+Hatfield%2DDodds%2C+D%2E)
9. P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, M. Rosenberg, X. Song, A. Stoica, S. Tiwary, and T. Wang. Ms marco: A human generated machine reading comprehension dataset, 2018. URL [https://arxiv.org/](https://arxiv.org/) abs/1611.09268. [↗](https://arxiv.org/) [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)
10. M. Bhatt, S. Chennabasappa, C. Nikolaidis, S. Wan, I. Evtimov, D. Gabi, D. Song, F. Ahmad, C. Aschermann, L. Fontana, S. Frolov, R. P. Giri, D. Kapil, Y. Kozyrakis, D. LeBlanc, J. Milazzo, A. Straumann, G. Synnaeve, V. Vontimitta, S. Whitman, and J. Saxe. Purple llama cyberseceval: A secure coding b… [find](https://scholar.google.com/scholar?q=M%2E+Bhatt%2C+S%2E+Chennabasappa%2C+C%2E+Nikolaidis%2C+S%2E+Wan%2C+I%2E+Evtimov%2C+D%2E+Gabi%2C+D%2E+Song%2C+F%2E+Ahmad%2C+C%2E+Aschermann%2C+L%2E+Fontana%2C+S%2E+Frolov%2C+R%2E+P%2E+Giri%2C+D%2E+Kapil%2C+Y%2E+Kozyrakis%2C+D%2E+LeBlanc%2C+J%2E+Milazzo%2C+A%2E+Strauman)
11. J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, L. Weng, and A. M adry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2024. URL https://arxiv.org/abs/2410.07095. [↗](https://arxiv.org/abs/2410.07095) [our card](https://md.research.elamur.ai/free/en/papers/chan-2024-mle-bench-evaluating-machine/)
12. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… [find](https://scholar.google.com/scholar?q=M%2E+Chen%2C+J%2E+Tworek%2C+H%2E+Jun%2C+Q%2E+Yuan%2C+H%2E+P%2E+de+Oliveira+Pinto%2C+J%2E+Kaplan%2C+H%2E+Edwards%2C+Y%2E+Burda%2C+N%2E+Joseph%2C+G%2E+Brockman%2C+A%2E+Ray%2C+R%2E+Puri%2C+G%2E+Krueger%2C+M%2E+Petrov%2C+H%2E+Khlaaf%2C+G%2E+Sastry%2C+P%2E+Mishkin%2C+B%2E+Chan) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
13. F. Chollet, M. Knoop, G. Kamradt, and B. Landers. Arc prize 2024: Technical report, 2024. URL https://arxiv.org/abs/2412.04604. [↗](https://arxiv.org/abs/2412.04604)
14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168. [↗](https://arxiv.org/abs/2110.14168) [our card](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/)
15. DeepSeek-AI. Deepseek-v3 technical report, 2024. URL [https://github.com/](https://github.com/) deepseek-ai/DeepSeek-V3/blob/main/DeepSeek_V3.pdf. [↗](https://github.com/)
16. D. Dua, Y. Wang, P. Dasigi, G. Stanovsky, S. Singh, and M. Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs, 2019. URL https://arxiv.org/abs/1903.00161. [↗](https://arxiv.org/abs/1903.00161) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/)
17. A. Dubey et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407. 21783. [↗](https://arxiv.org/abs/2407)
18. B. Gao, F. Song, Z. Yang, Z. Cai, Y. Miao, Q. Dong, L. Li, C. Ma, L. Chen, R. Xu, Z. Tang, B. Wang, D. Zan, S. Quan, G. Zhang, L. Sha, Y. Zhang, X. Ren, T. Liu, and B. Chang. Omni-math: A universal olympiad level mathematic benchmark for large language models, 2024. URL https://arxiv.org/abs/2410.07… [↗](https://arxiv.org/abs/2410.07…) [our card](https://md.research.elamur.ai/free/en/papers/gao-2024-omni-math-universal-olympiad-level/)
19. E. Glazer, E. Erdil, T. Besiroglu, D. Chicharro, E. Chen, A. Gunning, C. F. Olsson, J.-S. Denain, A. Ho, E. de Oliveira Santos, O. Järviniemi, M. Barnett, R. Sandler, J. Sevilla, Q. Ren, E. Pratt, L. Levine, G. Barkley, N. Stewart, B. Grechuk, T. Grechuk, and S. V. Enugandla. Frontiermath: A benchma… [find](https://scholar.google.com/scholar?q=E%2E+Glazer%2C+E%2E+Erdil%2C+T%2E+Besiroglu%2C+D%2E+Chicharro%2C+E%2E+Chen%2C+A%2E+Gunning%2C+C%2E+F%2E+Olsson%2C+J%2E%2DS%2E+Denain%2C+A%2E+Ho%2C+E%2E+de+Oliveira+Santos%2C+O%2E+J%C3%A4rviniemi%2C+M%2E+Barnett%2C+R%2E+Sandler%2C+J%2E+Sevilla%2C+Q%2E+Ren%2C+E%2E+Pratt%2C+L%2E+)
20. C. He, R. Luo, Y. Bai, S. Hu, Z. L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, J. Liu, L. Qi, Z. Liu, and M. Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2402.14008. [↗](https://arxiv.org/) [our card](https://md.research.elamur.ai/free/en/papers/he-2024-olympiadbench-challenging-benchmark-promoting-agi/)
21. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps, 2021. URL https://arxiv.org/abs/2105.09938. [↗](https://arxiv.org/abs/2105.09938) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/)
22. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009. 03300. [↗](https://arxiv.org/abs/2009) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/)
23. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874. [↗](https://arxiv.org/abs/2103.03874) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/)
24. D. Hendrycks, A. Zou, M. Mazeika, L. Tang, B. Li, D. Song, and J. Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures, 2022. URL https://arxiv.org/abs/2112.05135. — [arXiv:2112.05135](https://arxiv.org/abs/2112.05135) [↗](https://arxiv.org/abs/2112.05135)
25. A. Hosseini, A. Sordoni, D. Toyama, A. Courville, and R. Agarwal. Not all llm reasoners are created equal, 2024. URL https://arxiv.org/abs/2410.01748. — [arXiv:2410.01748](https://arxiv.org/abs/2410.01748) [↗](https://arxiv.org/abs/2410.01748)
26. A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. W. andMadhu Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, R. Sims, Z. Zhang, S. Goldshtein, and D.… [find](https://scholar.google.com/scholar?q=A%2E+Jacovi%2C+A%2E+Wang%2C+C%2E+Alberti%2C+C%2E+Tao%2C+J%2E+Lipovetz%2C+K%2E+Olszewska%2C+L%2E+Haas%2C+M%2E+Liu%2C+N%2E+Keating%2C+A%2E+Bloniarz%2C+C%2E+Saroufim%2C+C%2E+Fry%2C+D%2E+Marcus%2C+D%2E+Kukliansky%2C+G%2E+S%2E+Tomar%2C+J%2E+Swirhun%2C+J%2E+Xing%2C+L%2E+W%2E+andM)
27. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2310.06770. [↗](https://arxiv.org/) [our card](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/)
28. D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking benchmarking in nlp, 2021. URL https://arxiv.org/abs/2104.14337. [↗](https://arxiv.org/abs/2104.14337)
29. P. Kumar, E. Lau, S. Vijayakumar, T. Trinh, S. R. Team, E. Chang, V. Robinson, S. Hendryx, S. Zhou, M. Fredrikson, S. Yue, and Z. Wang. Refusal-trained llms are easily jailbroken as browser agents, 2024. URL https://arxiv.org/abs/2410.13886. — [arXiv:2410.13886](https://arxiv.org/abs/2410.13886) [↗](https://arxiv.org/abs/2410.13886)
30. J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Pon-napati, A. D. White, and S. G. Rodriques. Lab-bench: Measuring capabilities of language models for biology research, 2024. URL https://arxiv.org/abs/2407.10362. [↗](https://arxiv.org/abs/2407.10362)
31. N. Li, A. Pan, A. Gopal, S. Yue, D. Berrios, A. Gatti, J. D. Li, A.-K. Dombrowski, S. Goel, L. Phan, G. Mukobi, N. Helm-Burger, R. Lababidi, L. Justen, A. B. Liu, M. Chen, I. Barrass, O. Zhang, X. Zhu, R. Tamirisa, B. Bharathi, A. Khoja, Z. Zhao, A. Herbert-Voss, C. B. Breuer, S. Marks, O. Patel, A.… [find](https://scholar.google.com/scholar?q=N%2E+Li%2C+A%2E+Pan%2C+A%2E+Gopal%2C+S%2E+Yue%2C+D%2E+Berrios%2C+A%2E+Gatti%2C+J%2E+D%2E+Li%2C+A%2E%2DK%2E+Dombrowski%2C+S%2E+Goel%2C+L%2E+Phan%2C+G%2E+Mukobi%2C+N%2E+Helm%2DBurger%2C+R%2E+Lababidi%2C+L%2E+Justen%2C+A%2E+B%2E+Liu%2C+M%2E+Chen%2C+I%2E+Barrass%2C+O%2E+Zhang%2C+X%2E+Zhu)
32. P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024. URL https://arxiv.org/abs/2310.02255. [↗](https://arxiv.org/abs/2310.02255) [our card](https://md.research.elamur.ai/free/en/papers/lu-2023-mathvista-evaluating-mathematical-reasoning-foundation/)
33. T. R. McIntosh, T. Susnjak, N. Arachchilage, T. Liu, P. Watters, and M. N. Halgamuge. Inadequacies of large language model benchmarks in the era of generative artificial intelligence, 2024. URL https://arxiv.org/abs/2402.09880. — [arXiv:2402.09880](https://arxiv.org/abs/2402.09880) [↗](https://arxiv.org/abs/2402.09880)
34. Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial nli: A new benchmark for natural language understanding, 2020. URL https://arxiv.org/abs/1910. 14599. Openai o1 system card, 2024. [↗](https://arxiv.org/abs/1910)
35. OpenAI. o1-system-card-20240917.pdf. URL [https://cdn.openai.com/](https://cdn.openai.com/) [↗](https://cdn.openai.com/)
36. OpenAI. Openai and los alamos national laboratory announce bioscience research partnership, 2024. URL [https://openai.com/index/](https://openai.com/index/) openai-and-los-alamos-national-laboratory-work-together/. [↗](https://openai.com/index/)
37. OpenAI. Introducing swe-bench verified, 2024. introducing-swe-bench-verified/. URL [https://openai.com/index/](https://openai.com/index/) [↗](https://openai.com/index/)
38. OpenAI et al. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774. [↗](https://arxiv.org/abs/2303.08774) [our card](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
39. S. Ott, A. Barbosa-Silva, K. Blagec, J. Brauner, and M. Samwald. Mapping global dynamics of benchmark creation and saturation in artificial intelligence. Nature Communications, 13(1): 6793, 2022. [find](https://scholar.google.com/scholar?q=S%2E+Ott%2C+A%2E+Barbosa%2DSilva%2C+K%2E+Blagec%2C+J%2E+Brauner%2C+and+M%2E+Samwald%2E+Mapping+global+dynamics+of+benchmark+creation+and+saturation+in+artificial+intelligence%2E+Nature+Communications%2C+13%281%29%3A+6793%2C+2022%2E)
40. D. Owen. How predictable is language model benchmark performance?, 2024. URL https://arxiv.org/abs/2401.04757. [↗](https://arxiv.org/abs/2401.04757)
41. E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. M… [find](https://scholar.google.com/scholar?q=E%2E+Perez%2C+S%2E+Ringer%2C+K%2E+Luko%C5%A1iu%CC%84te%CC%87%2C+K%2E+Nguyen%2C+E%2E+Chen%2C+S%2E+Heiner%2C+C%2E+Pettit%2C+C%2E+Olsson%2C+S%2E+Kundu%2C+S%2E+Kadavath%2C+A%2E+Jones%2C+A%2E+Chen%2C+B%2E+Mann%2C+B%2E+Israel%2C+B%2E+Seethor%2C+C%2E+McKinnon%2C+C%2E+Olah%2C+D%2E+Yan%2C+D%2E+Amo) [our card](https://md.research.elamur.ai/free/en/papers/perez-2022-discovering-language-model-behaviors-model/)
42. M. Phuong, M. Aitchison, E. Catt, S. Cogan, A. Kaskasoli, V. Krakovna, D. Lindner, M. Rahtz, Y. Assael, S. Hodkinson, H. Howard, T. Lieberum, R. Kumar, M. A. Raad, A. Webson, L. Ho, S. Lin, S. Farquhar, M. Hutter, G. Deletang, A. Ruoss, S. El-Sayed, S. Brown, A. Dragan, R. Shah, A. Dafoe, and T. She… [find](https://scholar.google.com/scholar?q=M%2E+Phuong%2C+M%2E+Aitchison%2C+E%2E+Catt%2C+S%2E+Cogan%2C+A%2E+Kaskasoli%2C+V%2E+Krakovna%2C+D%2E+Lindner%2C+M%2E+Rahtz%2C+Y%2E+Assael%2C+S%2E+Hodkinson%2C+H%2E+Howard%2C+T%2E+Lieberum%2C+R%2E+Kumar%2C+M%2E+A%2E+Raad%2C+A%2E+Webson%2C+L%2E+Ho%2C+S%2E+Lin%2C+S%2E+Farquha)
43. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text, 2016. URL https://arxiv.org/abs/1606.05250. [↗](https://arxiv.org/abs/1606.05250) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
44. P. Rajpurkar, R. Jia, and P. Liang. Know what you don’t know: Unanswerable questions for squad, 2018. URL https://arxiv.org/abs/1806.03822. [↗](https://arxiv.org/abs/1806.03822) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
45. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023. URL https://arxiv.org/abs/2311.12022. [↗](https://arxiv.org/abs/2311.12022) [our card](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/)
46. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, et al. Large language models encode clinical knowledge. Nature, 620 (7972):172–180, 2023. [find](https://scholar.google.com/scholar?q=K%2E+Singhal%2C+S%2E+Azizi%2C+T%2E+Tu%2C+S%2E+S%2E+Mahdavi%2C+J%2E+Wei%2C+H%2E+W%2E+Chung%2C+N%2E+Scales%2C+A%2E+Tanwani%2C+H%2E+Cole%2DLewis%2C+S%2E+Pfohl%2C+et+al%2E+Large+language+models+encode+clinical+knowledge%2E+Nature%2C+620+%287972%29%3A172%E2%80%93180%2C+202) [our card](https://md.research.elamur.ai/free/en/papers/singhal-2022-large-language-models-encode-clinical/)
47. M. Skarlinski, J. Laurent, A. Bou, and A. White. About 30% of Humanity’s Last Exam chemistry/biology answers are likely wrong, July 2025. URL [https://www.futurehouse.org/research-announcements/hle-exam](https://www.futurehouse.org/research-announcements/hle-exam). [↗](https://www.futurehouse.org/research-announcements/hle-exam)
48. V. K. Srinivasan, Z. Dong, B. Zhu, B. Yu, H. Mao, D. Mosk-Aoyama, K. Keutzer, J. Jiao, and J. Zhang. Nexusraven: A commercially-permissive language model for function calling. In NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023. URL [https://openreview.net/forum?id=5lcPe6DqfI](https://openreview.net/forum?id=5lcPe6DqfI). [↗](https://openreview.net/forum?id=5lcPe6DqfI)
49. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slon… [find](https://scholar.google.com/scholar?q=A%2E+Srivastava%2C+A%2E+Rastogi%2C+A%2E+Rao%2C+A%2E+A%2E+M%2E+Shoeb%2C+A%2E+Abid%2C+A%2E+Fisch%2C+A%2E+R%2E+Brown%2C+A%2E+Santoro%2C+A%2E+Gupta%2C+A%2E+Garriga%2DAlonso%2C+A%2E+Kluska%2C+A%2E+Lewkowycz%2C+A%2E+Agarwal%2C+A%2E+Power%2C+A%2E+Ray%2C+A%2E+Warstadt%2C+A%2E+W%2E+Ko) [our card](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
50. S. A. Taghanaki, A. Khani, and A. Khasahmadi. Mmlu-pro+: Evaluating higher-order reasoning and shortcut learning in llms, 2024. URL https://arxiv.org/abs/2409.02257. — [arXiv:2409.02257](https://arxiv.org/abs/2409.02257) [↗](https://arxiv.org/abs/2409.02257)
51. G. Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL https://arxiv.org/abs/2403.05530. [↗](https://arxiv.org/abs/2403.05530) [our card](https://md.research.elamur.ai/free/en/papers/gemini-2024-team-gemini-1-5-unlocking-multimodal/)
52. G. Tsoukalas, J. Lee, J. Jennings, J. Xin, M. Ding, M. Jennings, A. Thakur, and S. Chaudhuri. Putnambench: Evaluating neural theorem-provers on the putnam mathematical competition, 2024. URL https://arxiv.org/abs/2407.11214. [↗](https://arxiv.org/abs/2407.11214)
53. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL https://arxiv.org/abs/1804.07461. [↗](https://arxiv.org/abs/1804.07461) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/)
54. A. Wang, Y. Pruksachatkun, N. Nangia, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems, 2020. URL https://arxiv.org/abs/1905.00537. [↗](https://arxiv.org/abs/1905.00537) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/)
55. Y. Wang, X. Ma, G. Zhang, Y. Ni, A. Chandra, S. Guo, W. Ren, A. Arulraj, X. He, Z. Jiang, T. Li, M. Ku, K. Wang, A. Zhuang, R. Fan, X. Yue, and W. Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (published at neurips 2024 track datasets and benchmarks), 2024… [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+X%2E+Ma%2C+G%2E+Zhang%2C+Y%2E+Ni%2C+A%2E+Chandra%2C+S%2E+Guo%2C+W%2E+Ren%2C+A%2E+Arulraj%2C+X%2E+He%2C+Z%2E+Jiang%2C+T%2E+Li%2C+M%2E+Ku%2C+K%2E+Wang%2C+A%2E+Zhuang%2C+R%2E+Fan%2C+X%2E+Yue%2C+and+W%2E+Chen%2E+Mmlu%2Dpro%3A+A+more+robust+and+challenging+multi%2Dt) [our card](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/)
56. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models, 2024. URL [https://arxiv.org/](https://arxiv.org/) abs/2411.04368. [↗](https://arxiv.org/) [our card](https://md.research.elamur.ai/free/en/papers/wei-2024-measuring-short-form-factuality-large/)
57. H. Wijk, T. Lin, J. Becker, S. Jawhar, N. Parikh, T. Broadley, L. Chan, M. Chen, J. Clymer, J. Dhyani, E. Ericheva, K. Garcia, B. Goodrich, N. Jurkovic, M. Kinniment, A. Lajko, S. Nix, L. Sato, W. Saunders, M. Taran, B. West, and E. Barnes. Re-bench: Evaluating frontier ai r&d capabilities of langua… [find](https://scholar.google.com/scholar?q=H%2E+Wijk%2C+T%2E+Lin%2C+J%2E+Becker%2C+S%2E+Jawhar%2C+N%2E+Parikh%2C+T%2E+Broadley%2C+L%2E+Chan%2C+M%2E+Chen%2C+J%2E+Clymer%2C+J%2E+Dhyani%2C+E%2E+Ericheva%2C+K%2E+Garcia%2C+B%2E+Goodrich%2C+N%2E+Jurkovic%2C+M%2E+Kinniment%2C+A%2E+Lajko%2C+S%2E+Nix%2C+L%2E+Sato%2C+W%2E+Sau)
58. xAI. Grok-2 beta release, 2024. URL [https://x.ai/blog/grok-2](https://x.ai/blog/grok-2). [↗](https://x.ai/blog/grok-2)
59. F. Yan, H. Mao, C. C.-J. Ji, T. Zhang, S. G. Patil, I. Stoica, and J. E. Gonzalez. Berkeley function calling leaderboard. [https://gorilla.cs.berkeley.edu/blogs/8_berkeley_](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_) function_calling_leaderboard.html, 2024. [↗](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_)
60. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering, 2018. URL https://arxiv.org/abs/1809.09600. [↗](https://arxiv.org/abs/1809.09600) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/)
61. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan. τ -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv.org/abs/2406.12045. [↗](https://arxiv.org/abs/2406.12045) [our card](https://md.research.elamur.ai/free/en/papers/yao-2024-tau-bench-benchmark-tool/)
62. A. K. Zhang, N. Perry, R. Dulepet, J. Ji, J. W. Lin, E. Jones, C. Menders, G. Hussein, S. Liu, D. Jasper, P. Peetathawatchai, A. Glenn, V. Sivashankar, D. Zamoshchin, L. Glikbarg, D. Askar-yar, M. Yang, T. Zhang, R. Alluri, N. Tran, R. Sangpisit, P. Yiorkadjis, K. Osele, G. Raghupathi, D. Boneh, D.… — [arXiv:2408.08926](https://arxiv.org/abs/2408.08926) [find](https://scholar.google.com/scholar?q=A%2E+K%2E+Zhang%2C+N%2E+Perry%2C+R%2E+Dulepet%2C+J%2E+Ji%2C+J%2E+W%2E+Lin%2C+E%2E+Jones%2C+C%2E+Menders%2C+G%2E+Hussein%2C+S%2E+Liu%2C+D%2E+Jasper%2C+P%2E+Peetathawatchai%2C+A%2E+Glenn%2C+V%2E+Sivashankar%2C+D%2E+Zamoshchin%2C+L%2E+Glikbarg%2C+D%2E+Askar%2Dyar%2C+M%2E+Yan)
63. W. Zhong, R. Cui, Y. Guo, Y. Liang, S. Lu, Y. Wang, A. Saied, W. Chen, and N. Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023. URL https://arxiv.org/abs/2304.06364. [↗](https://arxiv.org/abs/2304.06364) [our card](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)