---
title: "Decentralized Arena — paper by Yin et al. (2026)"
description: "The authors propose Decentralized Arena (De-Arena), where every model under evaluation also acts as… 46 references, cited by 1 paper here."
authors: ["Yanbin Yin", "Kun Zhou", "Zhen Wang", "Xiangdong Zhang", "Yifei Shao", "Shibo Hao", "Yi Gu", "Jieyuan Liu", "Somanshu Singla", "Tianyang Liu", "Eric P. Xing", "Zhengzhong Liu", "Haojian Jin", "Zhiting Hu"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.1639/"
doi: "10.18653/v1/2026.acl-long.1639"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2026-decentralized-arena-democratic-scalable-automatic/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2026-decentralized-arena-democratic-scalable-automatic/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models

# Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models

**Authors:** Yanbin Yin, [Kun Zhou](https://md.research.elamur.ai/free/en/authors/kun-zhou/), [Zhen Wang](https://md.research.elamur.ai/free/en/authors/zhen-wang/), Xiangdong Zhang, Yifei Shao, [Shibo Hao](https://md.research.elamur.ai/free/en/authors/shibo-hao/), Yi Gu, Jieyuan Liu, [Somanshu Singla](https://md.research.elamur.ai/free/en/authors/somanshu-singla/), [Tianyang Liu](https://md.research.elamur.ai/free/en/authors/tianyang-liu/), [Eric P. Xing](https://md.research.elamur.ai/free/en/authors/eric-p-xing/), [Zhengzhong Liu](https://md.research.elamur.ai/free/en/authors/zhengzhong-liu/), Haojian Jin and [Zhiting Hu](https://md.research.elamur.ai/free/en/authors/zhiting-hu/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.1639/) · [PDF](https://aclanthology.org/2026.acl-long.1639.pdf) · [DOI 10.18653/v1/2026.acl-long.1639](https://doi.org/10.18653/v1/2026.acl-long.1639) · [OpenAlex](https://openalex.org/W4417302192)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Ranking the fast-growing population of LLMs is hard. Fixed-answer benchmarks such as MMLU saturate. Human-voted leaderboards like Chatbot Arena are slow and expensive. Using one strong model (e.g., GPT-4) as judge introduces that model's own biases. The authors propose Decentralized Arena (De-Arena), where every model under evaluation also acts as a judge in pairwise comparisons, so no single "authority" dominates.

Letting all models judge all pairs on all questions would cost O(n³k), so two components make the approach practical:
- **Incremental coarse-to-fine ranking:** each new model is roughly positioned by binary search, then refined by reranking within a local window. Judges are re-weighted with an Elo-based scheme.
- **Automatic question selection:** a small subset of questions is chosen whose rankings agree best with the majority, which allows new evaluation dimensions to be built without manual curation.

Across 66 LLMs and nine fine-grained dimensions, De-Arena reaches up to 97% correlation with Chatbot Arena at a cost comparable to existing benchmarks. This makes it a scalable, less biased alternative to human or single-judge evaluation.

## Key points
- **Ranking procedure:** it starts from 6 seed models ranked by full pairwise comparison. Further models are inserted one at a time, with coarse placement costing O(kn log n) instead of the naive O(n³k).
- **Cost of the alternative:** ranking every model pair exhaustively needs about 4× more judge calls than De-Arena's coarse-to-fine scheme.
- **Robustness to insertion order:** across five random shuffles on MT-Bench, Spearman correlation with Chatbot Arena averaged 0.957 with a standard deviation of 0.0019.
- **Question selection:** choosing 32 of the 80 MT-Bench questions, De-Arena's method beat Random, LLaMA-3-8B perplexity, Anchor point, and GPT-4-based selection. Random was the strongest of these baselines.
- **Limitations:**
  - Elo-based judge weighting helps, but group bias from models with similar training can persist.
  - The method is aimed at open-ended tasks.
  - Ranking quality depends on how diverse the model pool and question set are.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California San Diego](https://md.research.elamur.ai/free/en/universities/university-of-california-san-diego/) — University, San Diego, United States · 73 works · 51,948 citations
- [Mohamed bin Zayed University of Artificial Intelligence](https://md.research.elamur.ai/free/en/universities/mohamed-bin-zayed-university-of-artificial-intelligence/) — University, Abu Dhabi, United Arab Emirates · 124 works · 5,676 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yanbin Yin, Kun Zhou, Zhen Wang, Xiangdong Zhang, Yifei Shao, Shibo Hao, Yi Gu, Jieyuan Liu, Somanshu Singla, Tianyang Liu, Eric P. Xing, Zhengzhong Liu, Haojian Jin and Zhiting Hu. 2026. Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yin2026decentralized,
  title = {{Decentralized Arena: Towards Democratic and Scalable Automatic Evaluation of Language Models}},
  author = {Yanbin Yin, Kun Zhou, Zhen Wang, Xiangdong Zhang, Yifei Shao, Shibo Hao, Yi Gu, Jieyuan Liu, Somanshu Singla, Tianyang Liu, Eric P. Xing, Zhengzhong Liu, Haojian Jin and Zhiting Hu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1639},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1639}
}
```

- [← previous DEAR: Distributional Error-Aware Reliability for Robust Multimodal Sentiment Analysis with Missing Modalities](https://md.research.elamur.ai/free/en/papers/zou-2026-dear-distributional-error-aware-reliability/)
- [next → Deciphering Cultural Representations in Large Language Models via Sparse Autoencoders](https://md.research.elamur.ai/free/en/papers/zou-2026-deciphering-cultural-representations-large-language/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/)

## References

46 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. AI@Meta. 2024. Llama 3 model card. [find](https://scholar.google.com/scholar?q=AI%40Meta%2E+2024%2E+Llama+3+model+card%2E)
3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
4. Imène Brigui-Chtioui and Inès Saad. 2011. A multiagent approach for collective decision making in knowledge management. Group Decision and Negotiation, 20:19–37. [find](https://scholar.google.com/scholar?q=Im%C3%A8ne+Brigui%2DChtioui+and+In%C3%A8s+Saad%2E+2011%2E+A+multiagent+approach+for+collective+decision+making+in+knowledge+management%2E+Group+Decision+and+Negotiation%2C+20%3A19%E2%80%9337%2E)
5. Amrita Chakraborty and Arpan Kumar Kar. 2017. Swarm intelligence: A review of algorithms. Nature-inspired computing and optimization: Theory and applications, pages 475–494. [find](https://scholar.google.com/scholar?q=Amrita+Chakraborty+and+Arpan+Kumar+Kar%2E+2017%2E+Swarm+intelligence%3A+A+review+of+algorithms%2E+Nature%2Dinspired+computing+and+optimization%3A+Theory+and+applications%2C+pages+475%E2%80%93494%2E)
6. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E Gonzalez, and 1 others. 2024. [Chatbot arena: An open platform for evaluating llms by human preference](https://md.research.elamur.ai/free/en/papers/chiang-2024-chatbot-arena-open-platform-evaluating/). arXiv preprint arXiv:2403.04132.
7. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. [https://github.com/open-compass/](https://github.com/open-compass/) opencompass. [↗](https://github.com/open-compass/)
8. Mengge Du, Yuntian Chen, Zhongzheng Wang, Longfeng Nie, and Dongxiao Zhang. 2024. Large language models for automatic equation discovery of nonlinear dynamics. Physics of Fluids, 36(9). [find](https://scholar.google.com/scholar?q=Mengge+Du%2C+Yuntian+Chen%2C+Zhongzheng+Wang%2C+Longfeng+Nie%2C+and+Dongxiao+Zhang%2E+2024%2E+Large+language+models+for+automatic+equation+discovery+of+nonlinear+dynamics%2E+Physics+of+Fluids%2C+36%289%29%2E)
9. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2E)
10. Arpad E Elo. 1967. The proposed uscf rating system, its development, theory, and applications. Chess life, 22(8):242–247. [find](https://scholar.google.com/scholar?q=Arpad+E+Elo%2E+1967%2E+The+proposed+uscf+rating+system%2C+its+development%2C+theory%2C+and+applications%2E+Chess+life%2C+22%288%29%3A242%E2%80%93247%2E)
11. Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. [https://huggingface.co/spaces/open-llm-leaderboard/open_llm_](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_) leaderboard. [↗](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_)
12. Peter A Gloor. 2006. Swarm creativity: Competitive advantage through collaborative innovation networks. Oxford University Press. [find](https://scholar.google.com/scholar?q=Peter+A+Gloor%2E+2006%2E+Swarm+creativity%3A+Competitive+advantage+through+collaborative+innovation+networks%2E+Oxford+University+Press%2E)
13. Shashwat Goel, Joschka Struber, Ilze Amanda Auzina, Karuna K Chandra, Ponnurangam Kumaraguru, Douwe Kiela, Ameya Prabhu, Matthias Bethge, and Jonas Geiping. 2025. Great models think alike and this undermines ai oversight. arXiv preprint arXiv:2502.04313. — [arXiv:2502.04313](https://arxiv.org/abs/2502.04313) [find](https://scholar.google.com/scholar?q=Shashwat+Goel%2C+Joschka+Struber%2C+Ilze+Amanda+Auzina%2C+Karuna+K+Chandra%2C+Ponnurangam+Kumaraguru%2C+Douwe+Kiela%2C+Ameya+Prabhu%2C+Matthias+Bethge%2C+and+Jonas+Geiping%2E+2025%2E+Great+models+think+alike+and+this+und)
14. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, and 1 others. 2024. [Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/). Advances in Neural Informati…
15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300.
16. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8):1–79. [find](https://scholar.google.com/scholar?q=Xinyi+Hou%2C+Yanjie+Zhao%2C+Yue+Liu%2C+Zhou+Yang%2C+Kailong+Wang%2C+Li+Li%2C+Xiapu+Luo%2C+David+Lo%2C+John+Grundy%2C+and+Haoyu+Wang%2E+2024%2E+Large+language+models+for+software+engineering%3A+A+systematic+literature+review%2E)
17. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, and 1 others. 2024a. The biggen bench: A principled benchmark for fine-grained evaluation of language models with language models. arXiv preprint arXiv:2406.05761… [find](https://scholar.google.com/scholar?q=Seungone+Kim%2C+Juyoung+Suk%2C+Ji+Yong+Cho%2C+Shayne+Longpre%2C+Chaeeun+Kim%2C+Dongkeun+Yoon%2C+Guijin+Son%2C+Yejin+Cho%2C+Sheikh+Shafayat%2C+Jinheon+Baek%2C+and+1+others%2E+2024a%2E+The+biggen+bench%3A+A+principled+benchmark+)
18. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024b. [Prometheus 2: An open source language model specialized in evaluating other language models](https://md.research.elamur.ai/free/en/papers/kim-2024-prometheus-2-open-source-language/). arXiv preprint arXiv:2405.01535.
19. Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023a. [Generative judge for evaluating alignment](https://md.research.elamur.ai/free/en/papers/li-2023-generative-judge-evaluating-alignment/). arXiv preprint arXiv:2310.05470.
20. Ruosen Li, Teerth Patel, and Xinya Du. 2023b. Prd: Peer rank and discussion improve large language model based evaluations. arXiv preprint arXiv:2307.02762. [find](https://scholar.google.com/scholar?q=Ruosen+Li%2C+Teerth+Patel%2C+and+Xinya+Du%2E+2023b%2E+Prd%3A+Peer+rank+and+discussion+improve+large+language+model+based+evaluations%2E+arXiv+preprint+arXiv%3A2307%2E02762%2E)
21. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023c. Alpacaeval: An automatic evaluator of instruction-following models. [find](https://scholar.google.com/scholar?q=Xuechen+Li%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Rohan+Taori%2C+Ishaan+Gulrajani%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023c%2E+Alpacaeval%3A+An+automatic+evaluator+of+instruction%2Dfollowing+models)
22. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, and 1 others. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+and+1+others%2E+2022%2E+Holistic+evaluation+of+language+m)
23. Anthony Lin. 2019. Binary search algorithm. WikiJour-nal of Science, 2(1):1–13. [find](https://scholar.google.com/scholar?q=Anthony+Lin%2E+2019%2E+Binary+search+algorithm%2E+WikiJour%2Dnal+of+Science%2C+2%281%29%3A1%E2%80%9313%2E)
24. Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. 2024. [Wildbench: Benchmarking llms with challenging tasks from real users in the wild](https://md.research.elamur.ai/free/en/papers/lin-2024-wildbench-benchmarking-llms-challenging-tasks/). arXiv preprint arXiv:2406.04770.
25. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412)
26. Jinjie Ni, Fuzhao Xue, Xiang Yue, Yuntian Deng, Mahir Shah, Kabir Jain, Graham Neubig, and Yang You. 2024. [Mixeval: Deriving wisdom of the crowd from llm benchmark mixtures](https://md.research.elamur.ai/free/en/papers/ni-2024-mixeval-deriving-wisdom-crowd-llm/). arXiv preprint arXiv:2406.06565.
27. OpenAI. 2024a. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv:2410.21276.
28. Preprint, OpenAI. 2024b. [Openai o1 system card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). Preprint, arXiv:2412.16720.
29. Samuel J Paech. 2023. [Eq-bench: An emotional intelligence benchmark for large language models](https://md.research.elamur.ai/free/en/papers/paech-2023-eq-bench-emotional-intelligence-benchmark/). arXiv preprint arXiv:2312.06281.
30. Arjun Panickssery, Samuel Bowman, and Shi Feng. 2024. Llm evaluators recognize and favor their own generations. Advances in Neural Information Processing Systems, 37:68772–68802. [find](https://scholar.google.com/scholar?q=Arjun+Panickssery%2C+Samuel+Bowman%2C+and+Shi+Feng%2E+2024%2E+Llm+evaluators+recognize+and+favor+their+own+generations%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A68772%E2%80%9368802%2E)
31. Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204.00498. [find](https://scholar.google.com/scholar?q=Nitarshan+Rajkumar%2C+Raymond+Li%2C+and+Dzmitry+Bahdanau%2E+2022%2E+Evaluating+the+text%2Dto%2Dsql+capabilities+of+large+language+models%2E+arXiv+preprint+arXiv%3A2204%2E00498%2E)
32. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. [GPQA: A graduate-level google-proof q&a benchmark](https://md.research.elamur.ai/free/en/papers/rein-2023-gpqa-graduate-level-google-proof/). In First Conference on Language Modeling.
33. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. [NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singap…
34. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference labeling by large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. [find](https://scholar.google.com/scholar?q=Keita+Saito%2C+Akifumi+Wachi%2C+Koki+Wataoka%2C+and+Youhei+Akimoto%2E+2023%2E+Verbosity+bias+in+preference+labeling+by+large+language+models%2E+In+NeurIPS+2023+Workshop+on+Instruction+Tuning+and+Instruction+Follo)
35. Juho Salminen and 1 others. 2015. The role of collective intelligence in crowdsourcing innovation. [find](https://scholar.google.com/scholar?q=Juho+Salminen+and+1+others%2E+2015%2E+The+role+of+collective+intelligence+in+crowdsourcing+innovation%2E)
36. Stephanie Schoch, Diyi Yang, and Yangfeng Ji. 2020. “this is a problem, don’t you agree?” framing and bias in human evaluation for natural language generation. In Proceedings of the 1st Workshop on Evaluating NLG Evaluation, pages 10–16. [find](https://scholar.google.com/scholar?q=Stephanie+Schoch%2C+Diyi+Yang%2C+and+Yangfeng+Ji%2E+2020%2E+%E2%80%9Cthis+is+a+problem%2C+don%E2%80%99t+you+agree%3F%E2%80%9D+framing+and+bias+in+human+evaluation+for+natural+language+generation%2E+In+Proceedings+of+the+1st+Workshop+on+Ev)
37. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, and 1 others. 2023. [Towards understanding sycophancy in language models](https://md.research.elamur.ai/free/en/papers/sharma-2023-towards-understanding-sycophancy-language-models/). arXiv preprint arXiv:2310.13548.
38. Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. 2023. [Llm-planner: Few-shot grounded planning for embodied agents with large language models](https://md.research.elamur.ai/free/en/papers/song-2022-llm-planner-few-shot-grounded/). In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
39. James Surowiecki. 2004. The Wisdom of Crowds: Why the Many Are Smarter than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. Doubleday, New York. [find](https://scholar.google.com/scholar?q=James+Surowiecki%2E+2004%2E+The+Wisdom+of+Crowds%3A+Why+the+Many+Are+Smarter+than+the+Few+and+How+Collective+Wisdom+Shapes+Business%2C+Economies%2C+Societies%2C+and+Nations%2E+Doubleday%2C+New+York%2E)
40. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. [Galactica: A large language model for science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/). arXiv preprint arXiv:2211.09085.
41. Rajan Vivek, Kawin Ethayarajh, Diyi Yang, and Douwe Kiela. 2024. Anchor points: Benchmarking models with much fewer examples. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1576–1601, St. Julian’s, Malta.… [find](https://scholar.google.com/scholar?q=Rajan+Vivek%2C+Kawin+Ethayarajh%2C+Diyi+Yang%2C+and+Douwe+Kiela%2E+2024%2E+Anchor+points%3A+Benchmarking+models+with+much+fewer+examples%2E+In+Proceedings+of+the+18th+Conference+of+the+European+Chapter+of+the+Assoc)
42. Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, and 1 others. 2024. [Mmlu-pro: A more robust and challenging multi-task language understanding benchmark](https://md.research.elamur.ai/free/en/papers/wang-2024-mmlu-pro-more-robust-challenging/). arXiv preprint arXiv:2406.01574.
43. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, and 1 others. 2024. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314. [find](https://scholar.google.com/scholar?q=Colin+White%2C+Samuel+Dooley%2C+Manley+Roberts%2C+Arka+Pal%2C+Ben+Feuer%2C+Siddhartha+Jain%2C+Ravid+Shwartz%2DZiv%2C+Neel+Jain%2C+Khalid+Saifullah%2C+Siddartha+Naidu%2C+and+1+others%2E+2024%2E+Livebench%3A+A+challenging%2C+contami)
44. Dimension Length Header List Bold All MT-Bench Algebra Probability Geometry 0.933 0.909 0.934 0.932 0.914 0.909 0.930 0.934 0.911 0.909 0.929 0.933 0.897 0.911 0.929 0.932 0.932 0.906 0.935 0.924 *Table 8: Style control ablation study results across different dimensions. Each column represents a dif… — [Rethinking Pragmatics in Large Language Models: Towards Open-Ended Evaluation and Preference Tuning](https://md.research.elamur.ai/free/en/papers/wu-2024-rethinking-pragmatics-large-language-models/)
45. Window size 1 2 3 MT-Bench Math Corr↑ Judges↓ Corr↑ Judges↓ 0.957 0.953 0.955 521495 684460 892260 0.962 0.960 0.960 808074 1069615 1284068 [find](https://scholar.google.com/scholar?q=Window+size+1+2+3+MT%2DBench+Math+Corr%E2%86%91+Judges%E2%86%93+Corr%E2%86%91+Judges%E2%86%93+0%2E957+0%2E953+0%2E955+521495+684460+892260+0%2E962+0%2E960+0%2E960+808074+1069615+1284068)
46. Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. Pixiu: A large language model, instruction data and evaluation benchmark for finance. arXiv preprint arXiv:2306.05443. *Table 9: Hyperparameter tuning results about window size.* Fanjia Yan, H… — [arXiv:2306.05443](https://arxiv.org/abs/2306.05443) [find](https://scholar.google.com/scholar?q=Qianqian+Xie%2C+Weiguang+Han%2C+Xiao+Zhang%2C+Yanzhao+Lai%2C+Min+Peng%2C+Alejandro+Lopez%2DLira%2C+and+Jimin+Huang%2E+2023%2E+Pixiu%3A+A+large+language+model%2C+instruction+data+and+evaluation+benchmark+for+finance%2E+arXiv+)

*Updated 31 October 2026*