---
title: "WildBench — paper by Lin et al. (2024) — summary and ideas"
description: "Existing automatic LLM benchmarks poorly reflect what real users actually ask. 43 references, cited by 91 papers here."
authors: ["Bill Yuchen Lin", "Yuntian Deng", "Khyathi Chandu", "Faeze Brahman", "Abhilasha Ravichander", "Valentina Pyatkin", "Nouha Dziri", "Ronan Le Bras", "Yejin Choi"]
year: 2024
venue: "arXiv:2406.04770"
source: "https://arxiv.org/abs/2406.04770"
doi: "10.48550/arxiv.2406.04770"
arxiv: "2406.04770"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/lin-2024-wildbench-benchmarking-llms-challenging-tasks/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2024-wildbench-benchmarking-llms-challenging-tasks/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

# WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild

**Authors:** [Bill Yuchen Lin](https://md.research.elamur.ai/free/en/authors/bill-yuchen-lin/), [Yuntian Deng](https://md.research.elamur.ai/free/en/authors/yuntian-deng/), [Khyathi Chandu](https://md.research.elamur.ai/free/en/authors/khyathi-chandu/), [Faeze Brahman](https://md.research.elamur.ai/free/en/authors/faeze-brahman/), [Abhilasha Ravichander](https://md.research.elamur.ai/free/en/authors/abhilasha-ravichander/), [Valentina Pyatkin](https://md.research.elamur.ai/free/en/authors/valentina-pyatkin/), [Nouha Dziri](https://md.research.elamur.ai/free/en/authors/nouha-dziri/), [Ronan Le Bras](https://md.research.elamur.ai/free/en/authors/ronan-le-bras/) and [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) · **Year:** 2024 · **Venue:** arXiv:2406.04770 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2406.04770) · [PDF](https://arxiv.org/pdf/2406.04770) · [DOI 10.48550/arxiv.2406.04770](https://doi.org/10.48550/arxiv.2406.04770) · [arXiv 2406.04770](https://arxiv.org/abs/2406.04770) · [OpenAlex](https://openalex.org/W4399511950)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [91](https://md.research.elamur.ai/free/en/papers/lin-2024-wildbench-benchmarking-llms-challenging-tasks/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Existing automatic LLM benchmarks poorly reflect what real users actually ask. MT-bench is tiny, AlpacaEval is dominated by easy information-seeking prompts, and ArenaHard leans heavily toward coding. WildBench addresses this by drawing hard, naturally distributed tasks from WildChat, a log of roughly one million real human–chatbot conversations. The authors filtered by length, turn count, language, toxicity and near-duplication. Three strong LLMs then rated task difficulty, items judged easy were dropped, and human reviewers produced a final set of 1,024 tasks. Evaluation uses an LLM judge (e.g., GPT-4-Turbo) guided by per-task checklists and asked to reason step by step before deciding. There are two metrics. WB-Reward compares responses pairwise against three baselines of different strength and includes a length-bias correction. WB-Score rates single answers more cheaply. Both track Chatbot Arena human Elo more closely than competing benchmarks. This makes WildBench a cheap, transparent and interpretable proxy for human preference on realistic, demanding prompts.

## Key points
- **Data pipeline:**
  - Queries of 10–3,000 tokens, at most five turns, English only, non-toxic.
  - SentenceBERT deduplication at cosine similarity above 0.9, keeping one conversation per device.
  - Difficulty labelled by GPT-4-Turbo, Claude-3-Sonnet and Opus, then 1,500 sampled tasks reduced to 1,024 after human review.
- **Query length:** WildBench queries average about 978.5 characters, versus 406.4 for ArenaHard and 164.9 for AlpacaEval.
- **WB-Reward:**
  - Five-level verdicts (much/slightly better or worse, tie).
  - Slight wins become ties when the winner's answer exceeds the loser's by more than K characters.
- **WB-Score:** checklist-guided 1–10 rating of individual responses. A combined WB-Elo metric powers the leaderboard on Hugging Face (allenai/WildBench).
- **Pearson correlation with Chatbot Arena Elo on top models:**
  - WB-Reward 0.98 and WB-Score 0.95.
  - ArenaHard 0.91; AlpacaEval2.0 0.89 length-controlled and 0.87 raw.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

## Cite

Cite as: Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras and Yejin Choi. 2024. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild. arXiv:2406.04770.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{lin2024wildbench,
  title = {{WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild}},
  author = {Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan {Le Bras} and Yejin Choi},
  year = {2024},
  doi = {10.48550/arxiv.2406.04770},
  eprint = {2406.04770},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.04770}
}
```

- [← previous SMILE: Single-turn to Multi-turn Inclusive Language Expansion via ChatGPT for Mental Health Support](https://md.research.elamur.ai/free/en/papers/qiu-2024-smile-single-turn-multi-turn/)
- [next → AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling](https://md.research.elamur.ai/free/en/papers/zhan-2024-anygpt-unified-multimodal-llm-discrete/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

43 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudo…](https://md.research.elamur.ai/free/en/papers/ai-2024-yi-open-foundation-models-01/)
2. Anthropic. The claude 3 model family: Opus, sonnet, haiku. [https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3). pdf, 2024. [↗](https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3)
3. The BigBench authors. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). ArXiv, abs/2206.04615, 2022. URL [https://api.semanticscholar.org/CorpusID:263625818](https://api.semanticscholar.org/CorpusID:263625818). [↗](https://api.semanticscholar.org/CorpusID:263625818)
4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024. [find](https://scholar.google.com/scholar?q=Wei%2DLin+Chiang%2C+Lianmin+Zheng%2C+Ying+Sheng%2C+Anastasios+Nikolas+Angelopoulos%2C+Tianle+Li%2C+Dacheng+Li%2C+Hao+Zhang%2C+Banghua+Zhu%2C+Michael+Jordan%2C+Joseph+E%2E+Gonzalez%2C+and+Ion+Stoica%2E+Chatbot+arena%3A+An+open+pl)
5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
6. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). arXiv preprint arXiv:2305.14387, 2023.
7. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B%2E+Hashimoto%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2C+2024%2E)
8. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. [Bias and fairness in large language models: A survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/), 2024.
9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
10. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. [Measuring mathematical problem solving with the math dataset](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-mathematical-problem-solving-math/). arXiv preprint arXiv:2103.03874, 2021.
11. Giwon Hong, Aryo Pradipta Gema, Rohit Saxena, Xiaotang Du, Ping Nie, Yu Zhao, Laura Perez-Beltrachini, Max Ryabinin, Xuanli He, Clémentine Fourrier, and Pasquale Minervini. The hallucinations leaderboard – an open effort to measure hallucinations in large language models, 2024. [find](https://scholar.google.com/scholar?q=Giwon+Hong%2C+Aryo+Pradipta+Gema%2C+Rohit+Saxena%2C+Xiaotang+Du%2C+Ping+Nie%2C+Yu+Zhao%2C+Laura+Perez%2DBeltrachini%2C+Max+Ryabinin%2C+Xuanli+He%2C+Cl%C3%A9mentine+Fourrier%2C+and+Pasquale+Minervini%2E+The+hallucinations+leaderbo)
12. Dongfu Jiang, Yishan Li, Ge Zhang, Wenhao Huang, Bill Yuchen Lin, and Wenhu Chen. Tigerscore: Towards building explainable metric for all text generation tasks. Transactions on Machine Learning Research, 2023. [find](https://scholar.google.com/scholar?q=Dongfu+Jiang%2C+Yishan+Li%2C+Ge+Zhang%2C+Wenhao+Huang%2C+Bill+Yuchen+Lin%2C+and+Wenhu+Chen%2E+Tigerscore%3A+Towards+building+explainable+metric+for+all+text+generation+tasks%2E+Transactions+on+Machine+Learning+Resear)
13. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. [Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models](https://md.research.elamur.ai/free/en/papers/jiang-2024-wildteaming-scale-wild-jailbreaks-adversarially/). ArXiv, abs/2406.18510, 2024. URL [https://api.se…](https://api.se…) [↗](https://api.se…)
14. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). arXiv preprint arXiv:2405.01535, 2024.
15. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). ArXiv, abs/2205.11916, 2022. URL [https://api.semanticscholar.org/CorpusID:249017743](https://api.semanticscholar.org/CorpusID:249017743). [↗](https://api.semanticscholar.org/CorpusID:249017743)
16. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. [Cmmlu: Measuring massive multitask language understanding in chinese](https://md.research.elamur.ai/free/en/papers/li-2023-cmmlu-measuring-massive-multitask-language/). arXiv preprint arXiv:2306.09212, 2023a.
17. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024. URL [https://lmsys.org/blog/2024-04-19-arena-hard/](https://lmsys.org/blog/2024-04-19-arena-hard/). [↗](https://lmsys.org/blog/2024-04-19-arena-hard/)
18. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 2023b. [↗](https://github.com/tatsu-lab/alpaca_eval)
19. Bill Yuchen Lin, Chengsong Huang, Qian Liu, Wenda Gu, Sam Sommerer, and Xiang Ren. On grounded planning for embodied tasks with language models. ArXiv, abs/2209.00465, 2022. URL [https://api.semanticscholar.org/CorpusID:251979509](https://api.semanticscholar.org/CorpusID:251979509). — [arXiv:2209.00465](https://arxiv.org/abs/2209.00465) [↗](https://api.semanticscholar.org/CorpusID:251979509)
20. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Raghavi Chandu, Chandra Bhagavatula, and Yejin Choi. [The unlocking spell on base llms: Rethinking alignment via in-context learning](https://md.research.elamur.ai/free/en/papers/lin-2023-unlocking-spell-base-llms-rethinking/). ArXiv, abs/2312.01552, 2023. URL https://api. semanticscholar.org/CorpusID:26560…
21. Bill Yuchen Lin, Ronan Le Bras, and Yejin Choi. Zebralogic: Benchmarking the logical reasoning ability of language models, 2024. URL [https://hf.co/spaces/allenai/](https://hf.co/spaces/allenai/) ZebraLogicBench-Leaderboard. [↗](https://hf.co/spaces/allenai/)
22. [Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Yuxian Gu, Hangliang Ding, Kai Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Shengqi Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench: Evaluating l…](https://md.research.elamur.ai/free/en/papers/liu-2023-agentbench-evaluating-llms-agents/)
23. Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, and Bill Yuchen Lin. Wildvision: Evaluating vision-language models in the wild with human preferences. arXiv preprint arXiv:2406.11069, 2024. — [arXiv:2406.11069](https://arxiv.org/abs/2406.11069) [find](https://scholar.google.com/scholar?q=Yujie+Lu%2C+Dongfu+Jiang%2C+Wenhu+Chen%2C+William+Yang+Wang%2C+Yejin+Choi%2C+and+Bill+Yuchen+Lin%2E+Wildvision%3A+Evaluating+vision%2Dlanguage+models+in+the+wild+with+human+preferences%2E+arXiv+preprint+arXiv%3A2406%2E1106)
24. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. [Harmbench: A standardized evaluation framework for automated red teaming and robust refusal](https://md.research.elamur.ai/free/en/papers/mazeika-2024-harmbench-standardized-evaluation-framework-automated/), 2024.
25. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. 2024a. URL [https://api.semanticscholar.org/CorpusID:](https://api.semanticscholar.org/CorpusID:) 269983560. [↗](https://api.semanticscholar.org/CorpusID:)
26. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward, 2024b. [find](https://scholar.google.com/scholar?q=Yu+Meng%2C+Mengzhou+Xia%2C+and+Danqi+Chen%2E+Simpo%3A+Simple+preference+optimization+with+a+reference%2Dfree+reward%2C+2024b%2E)
27. Meta. Introducing Meta Llama 3: The most capable openly available LLM to date. [https://ai.meta.com/blog/meta-llama-3/](https://ai.meta.com/blog/meta-llama-3/), 2023. [↗](https://ai.meta.com/blog/meta-llama-3/)
28. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann André LeCun, and Thomas Scialom. [Gaia: a benchmark for general ai assistants](https://md.research.elamur.ai/free/en/papers/mialon-2023-gaia-benchmark-general-ai-assistants/). ArXiv, abs/2311.12983, 2023. URL [https://api.semanticscholar.org/CorpusID:265351664](https://api.semanticscholar.org/CorpusID:265351664). [↗](https://api.semanticscholar.org/CorpusID:265351664)
29. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. [Factscore: Fine-grained atomic evaluation of factual precision in long form text generation](https://md.research.elamur.ai/free/en/papers/min-2023-factscore-fine-grained-atomic-evaluation/). arXiv preprint arXiv:2305.14251, 2023.
30. Abhika Mishra, Akari Asai, Vidhisha Balachandran, Yizhong Wang, Graham Neubig, Yulia Tsvetkov, and Hannaneh Hajishirzi. [Fine-grained hallucination detection and editing for language models](https://md.research.elamur.ai/free/en/papers/mishra-2024-fine-grained-hallucination-detection-editing/), 2024.
31. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/), 2023.
32. Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. The shifted and the overlooked: A task-oriented investigation of user-GPT interactions. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL… [find](https://scholar.google.com/scholar?q=Siru+Ouyang%2C+Shuohang+Wang%2C+Yang+Liu%2C+Ming+Zhong%2C+Yizhu+Jiao%2C+Dan+Iter%2C+Reid+Pryzant%2C+Chenguang+Zhu%2C+Heng+Ji%2C+and+Jiawei+Han%2E+The+shifted+and+the+overlooked%3A+A+task%2Doriented+investigation+of+user%2DGPT+)
33. Arjun Panickssery, Samuel R. Bowman, and Shi Feng. Llm evaluators recognize and favor their own generations, 2024. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Samuel+R%2E+Bowman%2C+and+Shi+Feng%2E+Llm+evaluators+recognize+and+favor+their+own+generations%2C+2024%2E)
34. Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xuansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. [Infobench: Evaluating instruction following ability in large language models](https://md.research.elamur.ai/free/en/papers/qin-2024-infobench-evaluating-instruction-following-ability/), 2024.
35. Nils Reimers and Iryna Gurevych. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/), 2019.
36. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. [Gpqa: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/), 2023.
37. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheonbok Park, Kang Min Yoo, and Stella Biderman. [Kmmlu: Measuring massive multitask language understanding in korean](https://md.research.elamur.ai/free/en/papers/son-2025-kmmlu-measuring-massive-multitask-language/). arXiv preprint arXiv:2402.11548, 2024.
38. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
39. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… [find](https://scholar.google.com/scholar?q=Hugo+Touvron%2C+Louis+Martin%2C+Kevin+Stone%2C+Peter+Albert%2C+Amjad+Almahairi%2C+Yasmine+Babaei%2C+Nikolay+Bashlykov%2C+Soumya+Batra%2C+Prajjwal+Bhargava%2C+Shruti+Bhosale%2C+Dan+Bikel%2C+Lukas+Blecher%2C+Cristian+Canton+Fe)
40. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
41. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/), 2024…
42. Wenting Zhao, Xiang Ren, John Frederick Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. [Wildchat: 1m chatgpt interaction logs in the wild](https://md.research.elamur.ai/free/en/papers/zhao-2024-wildchat-1m-chatgpt-interaction-logs/). 2024. URL [https://api.semanticscholar.org/CorpusID:269390491](https://api.semanticscholar.org/CorpusID:269390491). [↗](https://api.semanticscholar.org/CorpusID:269390491)
43. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Lianmin+Zheng%2C+Wei%2DLin+Chiang%2C+Ying+Sheng%2C+Siyuan+Zhuang%2C+Zhanghao+Wu%2C+Yonghao+Zhuang%2C+Zi+Lin%2C+Zhuohan+Li%2C+Dacheng+Li%2C+Eric+Xing%2C+et+al%2E+Judging+llm%2Das%2Da%2Djudge+with+mt%2Dbench+and+chatbot+arena%2E+Advance)

*Updated 24 October 2026*