---
title: "Chatbot Arena — paper by Chiang et al. (2024)"
description: "As an alternative, they propose Chatbot Arena, an open website where anyone can pose their own… 52 references, cited by 446 papers here."
authors: ["Wei-Lin Chiang", "Lianmin Zheng", "Ying Sheng", "Anastasios Nikolas Angelopoulos", "Tianle Li", "Dacheng Li", "Hao Zhang", "Banghua Zhu", "Michael Jordan", "Joseph E. Gonzalez", "Ion Stoica"]
year: 2024
venue: "arXiv:2403.04132"
source: "https://arxiv.org/abs/2403.04132"
doi: "10.48550/arxiv.2403.04132"
arxiv: "2403.04132"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/) / Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

# Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

**Authors:** [Wei-Lin Chiang](https://md.research.elamur.ai/free/en/authors/wei-lin-chiang/), [Lianmin Zheng](https://md.research.elamur.ai/free/en/authors/lianmin-zheng/), [Ying Sheng](https://md.research.elamur.ai/free/en/authors/ying-sheng/), Anastasios Nikolas Angelopoulos, [Tianle Li](https://md.research.elamur.ai/free/en/authors/tianle-li/), [Dacheng Li](https://md.research.elamur.ai/free/en/authors/dacheng-li/), [Hao Zhang](https://md.research.elamur.ai/free/en/authors/hao-zhang/), [Banghua Zhu](https://md.research.elamur.ai/free/en/authors/banghua-zhu/), [Michael Jordan](https://md.research.elamur.ai/free/en/authors/michael-i-jordan/), [Joseph E. Gonzalez](https://md.research.elamur.ai/free/en/authors/joseph-e-gonzalez/) and [Ion Stoica](https://md.research.elamur.ai/free/en/authors/ion-gabriel-stoica/) · **Year:** 2024 · **Venue:** arXiv:2403.04132 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2403.04132) · [PDF](https://arxiv.org/pdf/2403.04132) · [DOI 10.48550/arxiv.2403.04132](https://doi.org/10.48550/arxiv.2403.04132) · [arXiv 2403.04132](https://arxiv.org/abs/2403.04132) · [OpenAlex](https://openalex.org/W4392616695)

**Citations:** 100 citations (OpenAlex, 2026-10-14) · [446](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

The authors show that conventional LLM benchmarks poorly reflect how models are actually used. These are fixed sets of closed-ended questions with reference answers: over time they leak into training data, become saturated, and do not account for human preferences in open-ended tasks. As an alternative, they propose Chatbot Arena, an open website where anyone can pose their own question and receive answers from two anonymous models. The user then votes for the better answer and only afterwards learns which models they were. From these pairwise comparisons, a ranking is built using statistical methods, in particular the Bradley–Terry model. The authors also developed an adaptive procedure for selecting model pairs so that the ranking stabilizes with fewer votes, as well as a procedure for detecting malicious users. Data analysis showed that crowdsourced prompts are sufficiently diverse and discriminate well between models, and that ordinary users' votes largely agree with expert judgments. The platform has become one of the most influential public LLM leaderboards. Limitations: the user base is skewed toward enthusiasts and researchers, and model safety is not evaluated.

## Key points

- From April 2023 to January 2024, about 240K votes were collected from more than 90K users. More than 50 models were compared, including GPT-4, Claude, Gemini, LLaMA and Mistral.
- The conversations span more than 100 languages, but English accounts for 77% and Chinese for 5%. On average, there are about 8K votes per model.
- Adaptive sampling is more efficient than random sampling: to reach the same precision, random selection requires 54% more data (6,800 vs. 4,400 samples), and at another precision level, 5% more.
- The anomalous-user detector was tested on 25 anomalous and 25 normal users. It identified about 90% of the anomalous users and correctly classified normal users as normal in 60–70% of cases.
- The authors promise to release a dataset of more than 100K pairwise votes. A demo is available at chat.lmsys.org.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 64 works

🧭 Topics (3): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [🗨️ Dialogue systems](https://md.research.elamur.ai/free/en/topics/dialogue-systems/) · 624 works

💡 Concepts (1): [💡 Chatbots](https://md.research.elamur.ai/free/en/concepts/chatbot/) · 54 works

## Cite

Cite as: Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{chiang2024chatbot,
  title = {{Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference}},
  author = {Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez and Ion Stoica},
  year = {2024},
  doi = {10.48550/arxiv.2403.04132},
  eprint = {2403.04132},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.04132}
}
```

- [← previous SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://md.research.elamur.ai/free/en/papers/yang-2024-swe-agent-agent-computer-interfaces/)
- [next → How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://md.research.elamur.ai/free/en/papers/chen-2024-far-gpt-4v-closing/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/)

## References

52 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. [find](https://scholar.google.com/scholar?q=Bai%2C+Y%2E%2C+Jones%2C+A%2E%2C+Ndousse%2C+K%2E%2C+Askell%2C+A%2E%2C+Chen%2C+A%2E%2C+Das%2DSarma%2C+N%2E%2C+Drain%2C+D%2E%2C+Fort%2C+S%2E%2C+Ganguli%2C+D%2E%2C+Henighan%2C+T%2E%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcement+learning+from+)
2. Boubdir, M., Kim, E., Ermis, B., Hooker, S., and Fadaee, M. Elo uncovered: Robustness and best practices in language model evaluation, 2023. [find](https://scholar.google.com/scholar?q=Boubdir%2C+M%2E%2C+Kim%2C+E%2E%2C+Ermis%2C+B%2E%2C+Hooker%2C+S%2E%2C+and+Fadaee%2C+M%2E+Elo+uncovered%3A+Robustness+and+best+practices+in+language+model+evaluation%2C+2023%2E)
3. Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [find](https://scholar.google.com/scholar?q=Bradley%2C+R%2E+A%2E+and+Terry%2C+M%2E+E%2E+Rank+analysis+of+incomplete+block+designs%3A+I%2E+the+method+of+paired+comparisons%2E+Biometrika%2C+39%283%2F4%29%3A324%E2%80%93345%2C+1952%2E)
4. Busa-Fekete, R., Huellermeier, E., and Szörényi, B. Preference-based rank elicitation using statistical models: The case of mallows. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp.… [find](https://scholar.google.com/scholar?q=Busa%2DFekete%2C+R%2E%2C+Huellermeier%2C+E%2E%2C+and+Sz%C3%B6r%C3%A9nyi%2C+B%2E+Preference%2Dbased+rank+elicitation+using+statistical+models%3A+The+case+of+mallows%2E+In+Xing%2C+E%2E+P%2E+and+Jebara%2C+T%2E+%28eds%2E%29%2C+Proceedings+of+the+31st+Inter)
5. Busa-Fekete, R., Huellermeier, E., and Szörényi, B. Preference-based rank elicitation using statistical models: The case of mallows. In Xing, E. P. and Jebara, T. (eds.), Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pp.… [find](https://scholar.google.com/scholar?q=Busa%2DFekete%2C+R%2E%2C+Huellermeier%2C+E%2E%2C+and+Sz%C3%B6r%C3%A9nyi%2C+B%2E+Preference%2Dbased+rank+elicitation+using+statistical+models%3A+The+case+of+mallows%2E+In+Xing%2C+E%2E+P%2E+and+Jebara%2C+T%2E+%28eds%2E%29%2C+Proceedings+of+the+31st+Inter)
6. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374, 2021.
7. Chernoff, H. Sequential Design of Experiments, pp. 345–360. Springer New York, New York, NY, 1992. ISBN 978-1-4612-4380-9. doi: 10.1007/978-1-4612-4380-9_27. URL https://doi.org/10.1007/978-1-4612-4380-9_27. — [doi:10.1007/978-1-4612-4380-9_27](https://doi.org/10.1007/978-1-4612-4380-9_27) [↗](https://doi.org/10.1007/978-1-4612-4380-9_27)
8. Chiang, C.-H. and Lee, H.-y. Can large language models be an alternative to human evaluations? In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15607–15631, Toronto, Canada, Ju… [find](https://scholar.google.com/scholar?q=Chiang%2C+C%2E%2DH%2E+and+Lee%2C+H%2E%2Dy%2E+Can+large+language+models+be+an+alternative+to+human+evaluations%3F+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+Okazaki%2C+N%2E+%28eds%2E%29%2C+Proceedings+of+the+61st+Annual+Meeting+of+the+Ass)
9. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168, 2021.
10. Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2023. [find](https://scholar.google.com/scholar?q=Cui%2C+G%2E%2C+Yuan%2C+L%2E%2C+Ding%2C+N%2E%2C+Yao%2C+G%2E%2C+Zhu%2C+W%2E%2C+Ni%2C+Y%2E%2C+Xie%2C+G%2E%2C+Liu%2C+Z%2E%2C+and+Sun%2C+M%2E+Ultrafeedback%3A+Boosting+language+models+with+high%2Dquality+feedback%2C+2023%2E)
11. DiCiccio, T. J. and Efron, B. Bootstrap confidence intervals. Statistical science, 11(3):189–228, 1996. [find](https://scholar.google.com/scholar?q=DiCiccio%2C+T%2E+J%2E+and+Efron%2C+B%2E+Bootstrap+confidence+intervals%2E+Statistical+science%2C+11%283%29%3A189%E2%80%93228%2C+1996%2E)
12. Durrett, R. Probability: theory and examples, volume 49. Cambridge university press, 2019. [find](https://scholar.google.com/scholar?q=Durrett%2C+R%2E+Probability%3A+theory+and+examples%2C+volume+49%2E+Cambridge+university+press%2C+2019%2E)
13. Elo, A. E. The proposed uscf rating system, its development, theory, and applications. Chess Life, 22(8):242– 247, 1967. [find](https://scholar.google.com/scholar?q=Elo%2C+A%2E+E%2E+The+proposed+uscf+rating+system%2C+its+development%2C+theory%2C+and+applications%2E+Chess+Life%2C+22%288%29%3A242%E2%80%93+247%2C+1967%2E)
14. Fisher, R. A. Statistical methods for research workers. Number 5. Oliver and Boyd, 1928. [find](https://scholar.google.com/scholar?q=Fisher%2C+R%2E+A%2E+Statistical+methods+for+research+workers%2E+Number+5%2E+Oliver+and+Boyd%2C+1928%2E)
15. Freedman, D. A. On the so-called “huber sandwich estimator”’ and “robust standard errors”’. The American Statistician, 60(4):299–302, 2006. [find](https://scholar.google.com/scholar?q=Freedman%2C+D%2E+A%2E+On+the+so%2Dcalled+%E2%80%9Chuber+sandwich+estimator%E2%80%9D%E2%80%99+and+%E2%80%9Crobust+standard+errors%E2%80%9D%E2%80%99%2E+The+American+Statistician%2C+60%284%29%3A299%E2%80%93302%2C+2006%2E)
16. Gemini, T., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. [Gemini: a family of highly capable multimodal models](https://md.research.elamur.ai/free/en/papers/gemini-team-2023-gemini-family-highly-capable-multimodal/). arXiv preprint arXiv:2312.11805, 2023.
17. Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel, P., Levine, S., and Song, D. Koala: A dialogue model for academic research. Blog post, April 2023. URL [https://bair.berkeley.edu/blog/2023/04/03/koala/](https://bair.berkeley.edu/blog/2023/04/03/koala/). [↗](https://bair.berkeley.edu/blog/2023/04/03/koala/)
18. Grootendorst, M. [Bertopic: Neural topic modeling with a class-based tf-idf procedure](https://md.research.elamur.ai/free/en/papers/grootendorst-2022-bertopic-neural-topic-modeling-class/). arXiv preprint arXiv:2203.05794, 2022.
19. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations, 2020.
20. Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon, J. Time-uniform chernoff bounds via nonnegative supermartingales. 2020. [find](https://scholar.google.com/scholar?q=Howard%2C+S%2E+R%2E%2C+Ramdas%2C+A%2E%2C+McAuliffe%2C+J%2E%2C+and+Sekhon%2C+J%2E+Time%2Duniform+chernoff+bounds+via+nonnegative+supermartingales%2E+2020%2E)
21. Huang, Y., Lin, Z., Liu, X., Gong, Y., Lu, S., Lei, F., Liang, Y., Shen, Y., Lin, C., Duan, N., et al. Competition-level problems are effective llm evaluators. arXiv preprint arXiv:2312.02143, 2023. [find](https://scholar.google.com/scholar?q=Huang%2C+Y%2E%2C+Lin%2C+Z%2E%2C+Liu%2C+X%2E%2C+Gong%2C+Y%2E%2C+Lu%2C+S%2E%2C+Lei%2C+F%2E%2C+Liang%2C+Y%2E%2C+Shen%2C+Y%2E%2C+Lin%2C+C%2E%2C+Duan%2C+N%2E%2C+et+al%2E+Competition%2Dlevel+problems+are+effective+llm+evaluators%2E+arXiv+preprint+arXiv%3A2312%2E02143%2C+2023%2E)
22. Huber, P. J. et al. The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the fifth Berkeley symposium on mathematical statistics and probability, volume 1, pp. 221–233. Berkeley, CA: University of California Press, 1967. [find](https://scholar.google.com/scholar?q=Huber%2C+P%2E+J%2E+et+al%2E+The+behavior+of+maximum+likelihood+estimates+under+nonstandard+conditions%2E+In+Proceedings+of+the+fifth+Berkeley+symposium+on+mathematical+statistics+and+probability%2C+volume+1%2C+pp%2E+)
23. Hunter, D. R. MM algorithms for generalized Bradley-Terry models. The Annals of Statistics, 32(1):384 – 406, 2004. doi: 10.1214/aos/1079120141. URL https://doi.org/10.1214/aos/1079120141. — [doi:10.1214/aos/1079120141](https://doi.org/10.1214/aos/1079120141) [↗](https://doi.org/10.1214/aos/1079120141)
24. Karimi, M. R., Gürel, N. M., Karlaš, B., Rausch, J., Zhang, C., and Krause, A. Online active model selection for pre-trained classifiers. In International Conference on Artificial Intelligence and Statistics, pp. 307–315. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Karimi%2C+M%2E+R%2E%2C+G%C3%BCrel%2C+N%2E+M%2E%2C+Karla%C5%A1%2C+B%2E%2C+Rausch%2C+J%2E%2C+Zhang%2C+C%2E%2C+and+Krause%2C+A%2E+Online+active+model+selection+for+pre%2Dtrained+classifiers%2E+In+International+Conference+on+Artificial+Intelligence+and+Sta)
25. Karpinska, M., Akoury, N., and Iyyer, M. The perils of using Mechanical Turk to evaluate open-ended text generation. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1265–1285, Online and Pun… [find](https://scholar.google.com/scholar?q=Karpinska%2C+M%2E%2C+Akoury%2C+N%2E%2C+and+Iyyer%2C+M%2E+The+perils+of+using+Mechanical+Turk+to+evaluate+open%2Dended+text+generation%2E+In+Moens%2C+M%2E%2DF%2E%2C+Huang%2C+X%2E%2C+Specia%2C+L%2E%2C+and+Yih%2C+S%2E+W%2E%2Dt%2E+%28eds%2E%29%2C+Proceedings+of+th)
26. Kiela, D., Bartolo, M., Nie, Y., Kaushik, D., Geiger, A., Wu, Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al. Dynabench: Rethinking benchmarking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te… [find](https://scholar.google.com/scholar?q=Kiela%2C+D%2E%2C+Bartolo%2C+M%2E%2C+Nie%2C+Y%2E%2C+Kaushik%2C+D%2E%2C+Geiger%2C+A%2E%2C+Wu%2C+Z%2E%2C+Vidgen%2C+B%2E%2C+Prasad%2C+G%2E%2C+Singh%2C+A%2E%2C+Ringshia%2C+P%2E%2C+et+al%2E+Dynabench%3A+Rethinking+benchmarking+in+nlp%2E+In+Proceedings+of+the+2021+Conferen)
27. Köpf, A., Kilcher, Y., von Rütte, D., Anagnostidis, S., Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stanley, O., Nagyfi, R., et al. Openassistant conversations– democratizing large language model alignment. arXiv preprint arXiv:2304.[07327, 2023](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/).
28. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Langley%2C+P%2E+Crafting+papers+on+machine+learning%2E+In+Langley%2C+P%2E+%28ed%2E%29%2C+Proceedings+of+the+17th+International+Conference+on+Machine+Learning+%28ICML+2000%29%2C+pp%2E+1207%E2%80%931216%2C+Stanford%2C+CA%2C+2000%2E+Morgan+Kaufm)
29. Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 2023. [↗](https://github.com/tatsu-lab/alpaca_eval)
30. Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097, 2022.
31. Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D., Yasunaga, M., Zhang, Y., Narayanan, D., Wu, Y., Kumar, A., et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022. [find](https://scholar.google.com/scholar?q=Liang%2C+P%2E%2C+Bommasani%2C+R%2E%2C+Lee%2C+T%2E%2C+Tsipras%2C+D%2E%2C+Soylu%2C+D%2E%2C+Yasunaga%2C+M%2E%2C+Zhang%2C+Y%2E%2C+Narayanan%2C+D%2E%2C+Wu%2C+Y%2E%2C+Kumar%2C+A%2E%2C+et+al%2E+Holistic+evaluation+of+language+models%2E+arXiv+preprint+arXiv%3A2211%2E09110%2C+20)
32. Lin, Z., Wang, Z., Tong, Y., Wang, Y., Guo, Y., Wang, Y., and Shang, J. [ToxicChat: Unveiling hidden challenges of toxicity detection in real-world user-AI conversation](https://md.research.elamur.ai/free/en/papers/lin-2023-toxicchat-unveiling-hidden-challenges-toxicity/). In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4694–4702…
33. Liu, T.-Y. et al. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval, 3(3): 225–331, 2009. [find](https://scholar.google.com/scholar?q=Liu%2C+T%2E%2DY%2E+et+al%2E+Learning+to+rank+for+information+retrieval%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%283%29%3A+225%E2%80%93331%2C+2009%2E)
34. McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction, 2020. [find](https://scholar.google.com/scholar?q=McInnes%2C+L%2E%2C+Healy%2C+J%2E%2C+and+Melville%2C+J%2E+Umap%3A+Uniform+manifold+approximation+and+projection+for+dimension+reduction%2C+2020%2E)
35. OpenAI. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774, 2023. arXiv preprint Oren, Y., Meister, N., Chatterji, N., Ladhak, F., and Hashimoto, T. B. Proving test set contamination in black box language models. arXiv preprint arXiv:2310.17623, 2023.
36. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
37. Ramdas, A., Grünwald, P., Vovk, V., and Shafer, G. Game-theoretic statistics and safe anytime-valid inference. Statistical Science, 38(4):576–601, 2023. [find](https://scholar.google.com/scholar?q=Ramdas%2C+A%2E%2C+Gr%C3%BCnwald%2C+P%2E%2C+Vovk%2C+V%2E%2C+and+Shafer%2C+G%2E+Game%2Dtheoretic+statistics+and+safe+anytime%2Dvalid+inference%2E+Statistical+Science%2C+38%284%29%3A576%E2%80%93601%2C+2023%2E)
38. Rao, P. V. and Kupper, L. L. Ties in paired-comparison experiments: A generalization of the bradley-terry model. Journal of the American Statistical Association, 62(317): 194–204, 1967. doi: 10.1080/01621459.1967.10482901. — [doi:10.1080/01621459.1967.10482901](https://doi.org/10.1080/01621459.1967.10482901) [find](https://scholar.google.com/scholar?q=Rao%2C+P%2E+V%2E+and+Kupper%2C+L%2E+L%2E+Ties+in+paired%2Dcomparison+experiments%3A+A+generalization+of+the+bradley%2Dterry+model%2E+Journal+of+the+American+Statistical+Association%2C+62%28317%29%3A+194%E2%80%93204%2C+1967%2E+doi%3A+10%2E1080%2F0)
39. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. [Beyond the imitation game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/). Transactions on Machine Learning Research, 2023.
40. Szörényi, B., Busa-Fekete, R., Paul, A., and Hüllermeier, E. Online rank elicitation for plackett-luce: A dueling bandits approach. In Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.… — [AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models](https://md.research.elamur.ai/free/en/papers/zhong-2024-agieval-human-centric-benchmark-evaluating/)
41. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288, 2023.
42. Vovk, V. and Wang, R. E-values: Calibration, combination and applications. The Annals of Statistics, 49(3):1736– 1754, 2021. [find](https://scholar.google.com/scholar?q=Vovk%2C+V%2E+and+Wang%2C+R%2E+E%2Dvalues%3A+Calibration%2C+combination+and+applications%2E+The+Annals+of+Statistics%2C+49%283%29%3A1736%E2%80%93+1754%2C+2021%2E)
43. Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Wang%2C+Y%2E%2C+Kordi%2C+Y%2E%2C+Mishra%2C+S%2E%2C+Liu%2C+A%2E%2C+Smith%2C+N%2E+A%2E%2C+Khashabi%2C+D%2E%2C+and+Hajishirzi%2C+H%2E+Self%2Dinstruct%3A+Aligning+language+models+with+self%2Dgenerated+instructions%2E+In+Rogers%2C+A%2E%2C+Boyd%2DGraber%2C+J%2E%2C+and+O)
44. Waudby-Smith, I. and Ramdas, A. Estimating means of bounded random variables by betting. arXiv preprint arXiv:2010.09686, 2020. — [arXiv:2010.09686](https://arxiv.org/abs/2010.09686) [find](https://scholar.google.com/scholar?q=Waudby%2DSmith%2C+I%2E+and+Ramdas%2C+A%2E+Estimating+means+of+bounded+random+variables+by+betting%2E+arXiv+preprint+arXiv%3A2010%2E09686%2C+2020%2E+%E2%80%94+%5BarXiv%3A2010%2E09686%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2010%2E09686%29)
45. White, H. Maximum likelihood estimation of misspecified models. Econometrica: Journal of the econometric society, pp. 1–25, 1982. [find](https://scholar.google.com/scholar?q=White%2C+H%2E+Maximum+likelihood+estimation+of+misspecified+models%2E+Econometrica%3A+Journal+of+the+econometric+society%2C+pp%2E+1%E2%80%9325%2C+1982%2E)
46. Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and Stoica, I. [Rethinking benchmark and contamination for language models with rephrased samples](https://md.research.elamur.ai/free/en/papers/yang-2023-rethinking-benchmark-contamination-language-models/). arXiv preprint arXiv:2311.04850, 2023.
47. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791–4800, 2019. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%2C+Holtzman%2C+A%2E%2C+Bisk%2C+Y%2E%2C+Farhadi%2C+A%2E%2C+and+Choi%2C+Y%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Ling)
48. Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. [Lmsys-chat-1m: A large-scale real-world llm conversation dataset](https://md.research.elamur.ai/free/en/papers/zheng-2023-lmsys-chat-1m-large-scale/), 2023a.
49. Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., and Stoica, I. [Judging LLM-as-a-judge with MT-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Tra…
50. Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J. Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023. [find](https://scholar.google.com/scholar?q=Zhu%2C+B%2E%2C+Frick%2C+E%2E%2C+Wu%2C+T%2E%2C+Zhu%2C+H%2E%2C+and+Jiao%2C+J%2E+Starling%2D7b%3A+Improving+llm+helpfulness+%26+harmlessness+with+rlaif%2C+November+2023%2E)
51. Figure 8. Screenshot of Chatbot Arena. 1128 1256 1121 1212 848 970 1107 1014 966 768 729 825 813 786 779 651 786 439 442 549 650 476 627 548 216 405 359 312 245 205 216 278 324 296 266 293 199 222 326 327 384 535 482 372 301 489 412 393 357 290 184 186 380 368 326 337 418 323 246 309 382 425 428 396… [find](https://scholar.google.com/scholar?q=Figure+8%2E+Screenshot+of+Chatbot+Arena%2E+1128+1256+1121+1212+848+970+1107+1014+966+768+729+825+813+786+779+651+786+439+442+549+650+476+627+548+216+405+359+312+245+205+216+278+324+296+266+293+199+222+326)
52. Figure 9. The number of votes over time 3295 3201 2245 2252 2058 1587 1919 1832 1657 1632 1783 1855 1585 1770 1578 1758 1681 1529 1268 1571 1435 1556 1620 1573 1433 1435 1554 1656 1612 1491 1644 1252 1325 1233 1092 1200 1265 1056 1242 1143 1108 1204 874 951 705 724 [find](https://scholar.google.com/scholar?q=Figure+9%2E+The+number+of+votes+over+time+3295+3201+2245+2252+2058+1587+1919+1832+1657+1632+1783+1855+1585+1770+1578+1758+1681+1529+1268+1571+1435+1556+1620+1573+1433+1435+1554+1656+1612+1491+1644+1252+)

*Updated 31 October 2026*