---
title: "Instruction Mining — paper by Cao et al. (2023)"
description: "The authors propose InstructMining, an automatic pipeline that scores instruction–response pairs and… 38 references, cited by 72 papers here."
authors: ["Yihan Cao", "Yanbin Kang", "Chi Wang", "Lichao Sun"]
year: 2023
venue: "arXiv:2307.06290"
source: "https://arxiv.org/abs/2307.06290"
doi: "10.48550/arxiv.2307.06290"
arxiv: "2307.06290"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cao-2023-instruction-mining-instruction-data-selection/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cao-2023-instruction-mining-instruction-data-selection/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Instruction Mining: Instruction Data Selection for Tuning Large Language Models

# Instruction Mining: Instruction Data Selection for Tuning Large Language Models

**Authors:** Yihan Cao, Yanbin Kang, [Chi Wang](https://md.research.elamur.ai/free/en/authors/chi-wang/) and [Lichao Sun](https://md.research.elamur.ai/free/en/authors/lichao-sun/) · **Year:** 2023 · **Venue:** arXiv:2307.06290 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2307.06290) · [PDF](https://arxiv.org/pdf/2307.06290) · [DOI 10.48550/arxiv.2307.06290](https://doi.org/10.48550/arxiv.2307.06290) · [arXiv 2307.06290](https://arxiv.org/abs/2307.06290) · [OpenAlex](https://openalex.org/W4384268534)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/cao-2023-instruction-mining-instruction-data-selection/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Instruction tuning works best with high-quality data, but selecting that data usually requires expensive manual curation, as in LIMA. The authors propose InstructMining, an automatic pipeline that scores instruction–response pairs and picks a subset for finetuning LLMs. They use a finetuned model's loss on a curated, unbiased evaluation set as a stand-in for dataset quality. To avoid finetuning a model for every candidate dataset, they fit a least-squares linear model that predicts the log of this loss from cheap indicators. These include reward-model score, perplexity, MTLD lexical diversity, nearest-neighbour distance in SentenceBERT space, and UniEval dialogue scores. While varying subset size, they observe a double descent pattern: quality drives performance for small subsets, and quantity takes over past a threshold. They therefore apply BlendSearch from the FLAML library to find the best number of top-ranked examples. The resulting InstructMining-7B, built on LLaMA-2-7B, uses 2,532 of 100,000 examples and reaches state-of-the-art results on LLM-as-a-judge and the Huggingface Open LLM benchmark. This shows that principled selection can lower training cost while improving model quality.

## Key points
- The final rule combines four indicators: reward score from oasst-rm-pythia-1.4b, plus UniEval understandability, naturalness and coherence. In ablation, the reward score matters most: removing it raises loss by 0.030 on SELF-INSTRUCT and 0.051 on MT-BENCH.
- The optimal subset is about 2.5% of the pool (2,532 / 100,000). It lifts LLaMA-2-7B by 4.93 points on Open LLM and yields answers equal to or better than Vicuna-7B-v1.5 in 64.67% of comparisons.
- Double descent appears on OpenOrca: loss falls, rises after roughly 10,000 examples, then falls again. The quality-selected and random curves eventually cross and converge.
- In robustness tests, selected data beats random data with LLaMA-1-7B, LLaMA-2-13B and LoRA finetuning. The LoRA gain is marginal (SELF-INSTRUCT loss 1.0698 vs 1.0700).
- Open LLM evaluation follows the leaderboard setup: ARC 25-shot, HellaSwag 10-shot, MMLU 5-shot and TruthfulQA zero-shot.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Lehigh University](https://md.research.elamur.ai/free/en/universities/lehigh-university/) — University, Bethlehem, United States · 10 works · 1,573 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yihan Cao, Yanbin Kang, Chi Wang and Lichao Sun. 2023. Instruction Mining: Instruction Data Selection for Tuning Large Language Models. arXiv:2307.06290.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cao2023instruction,
  title = {{Instruction Mining: Instruction Data Selection for Tuning Large Language Models}},
  author = {Yihan Cao, Yanbin Kang, Chi Wang and Lichao Sun},
  year = {2023},
  doi = {10.48550/arxiv.2307.06290},
  eprint = {2307.06290},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2307.06290}
}
```

- [← previous Cumulative Reasoning with Large Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2023-cumulative-reasoning-large-language-models/)
- [next → Large Language Models as Analogical Reasoners](https://md.research.elamur.ai/free/en/papers/yasunaga-2023-large-language-models-analogical-reasoners/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Åke Björck. Least squares methods. Handbook of numerical analysis, 1:465–652, 1990. [find](https://scholar.google.com/scholar?q=A%CC%8Ake+Bjo%CC%88rck%2E+Least+squares+methods%2E+Handbook+of+numerical+analysis%2C+1%3A465%E2%80%93652%2C+1990%2E)
2. Alexander Bukharin and Tuo Zhao. Data diversity matters for robust instruction tuning. 2023. [find](https://scholar.google.com/scholar?q=Alexander+Bukharin+and+Tuo+Zhao%2E+Data+diversity+matters+for+robust+instruction+tuning%2E+2023%2E)
3. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. [Alpagasus: Training a better alpaca with fewer data](https://md.research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/), 2023.
4. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep)
5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457, 2018.
6. Wei Dong, Charikar Moses, and Kai Li. Efficient k-nearest neighbor graph construction for generic similarity measures. In Proceedings of the 20th international conference on World wide web, pp. 577–586, 2011. [find](https://scholar.google.com/scholar?q=Wei+Dong%2C+Charikar+Moses%2C+and+Kai+Li%2E+Efficient+k%2Dnearest+neighbor+graph+construction+for+generic+similarity+measures%2E+In+Proceedings+of+the+20th+international+conference+on+World+wide+web%2C+pp%2E+577%E2%80%9358)
7. Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels. 2024. [find](https://scholar.google.com/scholar?q=Logan+Engstrom%2C+Axel+Feldmann%2C+and+Aleksander+Madry%2E+Dsdm%3A+Model%2Daware+dataset+selection+with+datamodels%2E+2024%2E)
8. Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, September… — [doi:10.5281/zenodo.5371628](https://doi.org/10.5281/zenodo.5371628) [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Jonathan+Tow%2C+Stella+Biderman%2C+Sid+Black%2C+Anthony+DiPofi%2C+Charles+Foster%2C+Laurence+Golding%2C+Jeffrey+Hsu%2C+Kyle+McDonell%2C+Niklas+Muennighoff%2C+Jason+Phang%2C+Laria+Reynolds%2C+Eric+Tang%2C+Anish+Thite)
9. Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. [Demystifying prompts in language models via perplexity estimation](https://md.research.elamur.ai/free/en/papers/gonen-2022-demystifying-prompts-language-models-perplexity/), 2022.
10. Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023. [find](https://scholar.google.com/scholar?q=Suriya+Gunasekar%2C+Yi+Zhang%2C+Jyoti+Aneja%2C+Caio+Ce%CC%81sar+Teodoro+Mendes%2C+Allie+Del+Giorno%2C+Sivakanth+Gopi%2C+Mojan+Javaheripi%2C+Piero+Kauffmann%2C+Gustavo+de+Rosa%2C+Olli+Saarikivi%2C+et+al%2E+Textbooks+are+all+you+)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). arXiv preprint arXiv:2009.03300, 2020.
12. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/), 2022.
13. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. [Lora: Low-rank adaptation of large language models](https://md.research.elamur.ai/free/en/papers/hu-2021-lora-low-rank-adaptation-large/), 2021.
14. Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, Xian Li, Brian O’Horo, Gabriel Pereyra, Jeff Wang, Christopher Dewan, Asli Celikyilmaz, Luke Zettlemoyer, and Ves Stoyanov. Opt-iml: Scaling language m… — [OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization](https://md.research.elamur.ai/free/en/papers/iyer-2022-opt-iml-scaling-language-model/)
15. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Op… — [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/)
16. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and ”Teknium”. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca, 2023. [find](https://scholar.google.com/scholar?q=Wing+Lian%2C+Bleys+Goodson%2C+Eugene+Pentland%2C+Austin+Cook%2C+Chanvichet+Vong%2C+and+%E2%80%9DTeknium%E2%80%9D%2E+Openorca%3A+An+open+dataset+of+gpt+augmented+flan+reasoning+traces%2E+https%3A%2F%2Fhttps%3A%2F%2Fhuggingface%2Eco%2FOpen%2DOrca%2FOpenO)
17. Stephanie Lin, Jacob Hilton, and Owain Evans. [Truthfulqa: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). arXiv preprint arXiv:2109.07958, 2021.
18. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/), 2023.
19. Philip M McCarthy and Scott Jarvis. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381–392, 2010. [find](https://scholar.google.com/scholar?q=Philip+M+McCarthy+and+Scott+Jarvis%2E+Mtld%2C+vocd%2Dd%2C+and+hd%2Dd%3A+A+validation+study+of+sophisticated+approaches+to+lexical+diversity+assessment%2E+Behavior+research+methods%2C+42%282%29%3A381%E2%80%93392%2C+2010%2E)
20. Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: Where bigger models and more data hurt. Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003, 2021. [find](https://scholar.google.com/scholar?q=Preetum+Nakkiran%2C+Gal+Kaplun%2C+Yamini+Bansal%2C+Tristan+Yang%2C+Boaz+Barak%2C+and+Ilya+Sutskever%2E+Deep+double+descent%3A+Where+bigger+models+and+more+data+hurt%2E+Journal+of+Statistical+Mechanics%3A+Theory+and+Exp)
21. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
22. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). arXiv preprint arXiv:2304.03277, 2023.
23. Nils Reimers and Iryna Gurevych. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https://arxiv.org/abs/1908.10084. [↗](https://arxiv.org/abs/1908.10084)
24. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022. [find](https://scholar.google.com/scholar?q=Teven+Le+Scao%2C+Angela+Fan%2C+Christopher+Akiki%2C+Ellie+Pavlick%2C+Suzana+Ilic%CC%81%2C+Daniel+Hesslow%2C+Roman+Castagne%CC%81%2C+Alexandra+Sasha+Luccioni%2C+Franc%CC%A7ois+Yvon%2C+Matthias+Galle%CC%81%2C+et+al%2E+Bloom%3A+A+176b%2Dparameter+op)
25. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca), 2023. [↗](https://github.com/tatsu-lab/stanford_alpaca)
26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/), 2023.
27. Chi Wang, Qingyun Wu, Silu Huang, and Amin Saied. Economical hyperparameter optimization with blended search strategy. In ICLR’21, 2021a. [find](https://scholar.google.com/scholar?q=Chi+Wang%2C+Qingyun+Wu%2C+Silu+Huang%2C+and+Amin+Saied%2E+Economical+hyperparameter+optimization+with+blended+search+strategy%2E+In+ICLR%E2%80%9921%2C+2021a%2E)
28. Chi Wang, Qingyun Wu, Markus Weimer, and Erkang Zhu. Flaml: A fast and lightweight automl library, 2021b. [find](https://scholar.google.com/scholar?q=Chi+Wang%2C+Qingyun+Wu%2C+Markus+Weimer%2C+and+Erkang+Zhu%2E+Flaml%3A+A+fast+and+lightweight+automl+library%2C+2021b%2E)
29. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022a. [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Yeganeh+Kordi%2C+Swaroop+Mishra%2C+Alisa+Liu%2C+Noah+A+Smith%2C+Daniel+Khashabi%2C+and+Hannaneh+Hajishirzi%2E+Self%2Dinstruct%3A+Aligning+language+model+with+self+generated+instructions%2E+arXiv+preprint+)
30. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Ma… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+Eshaan+Pathak%2C+Giannis+Karaman)
31. Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. [How far can camels go? exploring the state of instruction tuning on open resources](https://md.research.elamur.ai/free/en/papers/wang-2023-far-camels-go-exploring-state/), 2023.
32. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). arXiv preprint arXiv:2109.01652, 2021.
33. Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. Self-evolved diverse data sampling for efficient instruction tuning. 2023. [find](https://scholar.google.com/scholar?q=Shengguang+Wu%2C+Keming+Lu%2C+Benfeng+Xu%2C+Junyang+Lin%2C+Qi+Su%2C+and+Chang+Zhou%2E+Self%2Devolved+diverse+data+sampling+for+efficient+instruction+tuning%2E+2023%2E)
34. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2C+2019%2E)
35. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models at evaluating instruction following. arXiv preprint arXiv:2310.07641, 2023. [find](https://scholar.google.com/scholar?q=Zhiyuan+Zeng%2C+Jiatong+Yu%2C+Tianyu+Gao%2C+Yu+Meng%2C+Tanya+Goyal%2C+and+Danqi+Chen%2E+Evaluating+large+language+models+at+evaluating+instruction+following%2E+arXiv+preprint+arXiv%3A2310%2E07641%2C+2023%2E)
36. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/), 2023.
37. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. [Towards a unified multi-dimensional evaluator for text generation](https://md.research.elamur.ai/free/en/papers/zhong-2022-unified-multi-dimensional-evaluator-text/), 2022.
38. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206, 2023. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2C+2023%2E)

*Updated 2 November 2026*