---
title: "Measuring Data Diversity for Instruction Tuning: A Systematic"
description: "They build 53 IT datasets of 10,000 samples each from a pool combining WizardLM, ShareGPT and… 59 references, cited by 6 papers here."
authors: ["Yuming Yang", "Yang Nan", "Junjie Ye", "Shihan Dou", "Xiao Wang", "Shuo Li", "Huijie Lv", "Tao Gui", "Qi Zhang", "Xuanjing Huang"]
year: 2025
venue: "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2025.acl-long.908/"
doi: "10.18653/v1/2025.acl-long.908"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yang-2025-measuring-data-diversity-instruction-tuning/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yang-2025-measuring-data-diversity-instruction-tuning/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/) / Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric

# Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric

**Authors:** [Yuming Yang](https://md.research.elamur.ai/free/en/authors/yuming-yang/), [Yang Nan](https://md.research.elamur.ai/free/en/authors/yang-nan/), [Junjie Ye](https://md.research.elamur.ai/free/en/authors/junjie-ye/), [Shihan Dou](https://md.research.elamur.ai/free/en/authors/shihan-dou/), [Xiao Wang](https://md.research.elamur.ai/free/en/authors/xiao-wang/), Shuo Li, Huijie Lv, [Tao Gui](https://md.research.elamur.ai/free/en/authors/tao-gui/), [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/) and [Xuanjing Huang](https://md.research.elamur.ai/free/en/authors/xuanjing-huang/) · **Year:** 2025 · **Venue:** [Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2025](https://md.research.elamur.ai/free/en/venues/acl/2025/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2025.acl-long.908/) · [PDF](https://aclanthology.org/2025.acl-long.908.pdf) · [DOI 10.18653/v1/2025.acl-long.908](https://doi.org/10.18653/v1/2025.acl-long.908) · [OpenAlex](https://openalex.org/W4412944993)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [6](https://md.research.elamur.ai/free/en/papers/yang-2025-measuring-data-diversity-instruction-tuning/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Diversity is widely seen as a key property of instruction-tuning (IT) data, yet there is no agreed way to quantify it. As a result, diversity-oriented data selection is largely guesswork and sometimes does worse than random sampling. The authors test 11 existing diversity measures, including lexical, distance-based and distribution-based ones. They build 53 IT datasets of 10,000 samples each from a pool combining WizardLM, ShareGPT and UltraChat, fine-tune LLMs on every dataset, and check how well each measure predicts benchmark results. Two lessons emerge. First, a good metric should focus on differences between nearby samples. Second, it should account for uneven information density, since dense areas such as math or code hold more unique content than semantic similarity alone suggests. Building on this, they propose NovelSum. It scores a dataset by summing each sample's "novelty", computed as proximity-weighted, density-aware distances to the other samples. NovelSum tracks model performance far better than earlier metrics. A greedy selector built on it, NovelSelect, beats existing selection strategies. This gives data engineers a validated objective to optimize.

## Key points
- The study covers 11 baseline metrics, such as TTR, vocd-D, DistSum, KNN distance, Vendi Score, Facility Location and Partition Entropy. These are tested against 53 datasets built with K-Center-Greedy, Repr Filter, QDIT, K-means, Random, Farthest and Duplicate strategies.
- Fine-tuning uses LLaMA-3-8B and Qwen-2.5-7B. Performance is evaluated on MT-bench and AlpacaEval, combined via Z-scores.
- NovelSum reaches Pearson r = 0.98 and Spearman r = 0.95, averaging about 0.97, which is higher than every prior metric.
- NovelSelect, a greedy method optimizing NovelSum, outperforms other diversity-based selectors when picking 10k samples. It is comparably cheap to run and can incorporate quality scores.
- Embeddings taken from the same model used as the fine-tuning backbone give the strongest correlations. The code is available at github.com/UmeanNever/NovelSum.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang and Xuanjing Huang. 2025. Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yang2025measuring,
  title = {{Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric}},
  author = {Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang and Xuanjing Huang},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.908},
  url = {https://doi.org/10.18653/v1/2025.acl-long.908}
}
```

- [← previous MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://md.research.elamur.ai/free/en/papers/liu-2025-mdcure-scalable-pipeline-multi-document/)
- [next → Memorization or Reasoning? Exploring the Idiom Understanding of LLMs](https://md.research.elamur.ai/free/en/papers/kim-2025-memorization-or-reasoning-exploring-idiom/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/8/)

## References

59 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Alexander Bukharin, Shiyang Li, Zhengyang Wang, Jingfeng Yang, Bing Yin, Xian Li, Chao Zhang, Tuo Zhao, and Haoming Jiang. 2024. Data diversity matters for robust instruction tuning. In Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024,… [find](https://scholar.google.com/scholar?q=Alexander+Bukharin%2C+Shiyang+Li%2C+Zhengyang+Wang%2C+Jingfeng+Yang%2C+Bing+Yin%2C+Xian+Li%2C+Chao+Zhang%2C+Tuo+Zhao%2C+and+Haoming+Jiang%2E+2024%2E+Data+diversity+matters+for+robust+instruction+tuning%2E+In+Findings+of+th)
3. Alfonso Cevallos, Friedrich Eisenbrand, and Rico Zen-klusen. 2017. Local search for max-sum diversification. In Proceedings of the Twenty-Eighth Annual ACM-SIAM Symposium on Discrete Algorithms, pages 130–142. SIAM. [find](https://scholar.google.com/scholar?q=Alfonso+Cevallos%2C+Friedrich+Eisenbrand%2C+and+Rico+Zen%2Dklusen%2E+2017%2E+Local+search+for+max%2Dsum+diversification%2E+In+Proceedings+of+the+Twenty%2DEighth+Annual+ACM%2DSIAM+Symposium+on+Discrete+Algorithms%2C+pages)
4. Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao. 2023a. Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning. arXiv preprint arXiv:2305.09246. — [arXiv:2305.09246](https://arxiv.org/abs/2305.09246) [find](https://scholar.google.com/scholar?q=Hao+Chen%2C+Yiming+Zhang%2C+Qi+Zhang%2C+Hantao+Yang%2C+Xiaomeng+Hu%2C+Xuetao+Ma%2C+Yifan+Yanggong%2C+and+Junbo+Zhao%2E+2023a%2E+Maybe+only+0%2E5%25+data+is+needed%3A+A+preliminary+exploration+of+low+training+data+instruction)
5. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. 2023b. [Alpagasus: Training a better alpaca with fewer data](https://md.research.elamur.ai/free/en/papers/chen-2023-alpagasus-training-better-alpaca-fewer/). arXiv preprint arXiv:2307.08701.
6. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023), 2(3):6. [↗](https://vicuna.lmsys)
7. Israel Cohen, Yiteng Huang, Jingdong Chen, Jacob Benesty, Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. Noise reduction in speech processing, pages 1–4. [find](https://scholar.google.com/scholar?q=Israel+Cohen%2C+Yiteng+Huang%2C+Jingdong+Chen%2C+Jacob+Benesty%2C+Jacob+Benesty%2C+Jingdong+Chen%2C+Yiteng+Huang%2C+and+Israel+Cohen%2E+2009%2E+Pearson+correlation+coefficient%2E+Noise+reduction+in+speech+processing%2C+pag)
8. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [find](https://scholar.google.com/scholar?q=Mike+Conover%2C+Matt+Hayes%2C+Ankit+Mathur%2C+Jianwei+Xie%2C+Jun+Wan%2C+Sam+Shah%2C+Ali+Ghodsi%2C+Patrick+Wendell%2C+Matei+Zaharia%2C+and+Reynold+Xin%2E+2023%2E+Free+dolly%3A+Introducing+the+world%E2%80%99s+first+truly+open+instruct)
9. Harshita Diddee and Daphne Ippolito. 2024. Chasing random: Instruction selection strategies fail to generalize. arXiv preprint arXiv:2410.15225. [find](https://scholar.google.com/scholar?q=Harshita+Diddee+and+Daphne+Ippolito%2E+2024%2E+Chasing+random%3A+Instruction+selection+strategies+fail+to+generalize%2E+arXiv+preprint+arXiv%3A2410%2E15225%2E)
10. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. [Enhancing chat language models by scaling high-quality instructional conversations](https://md.research.elamur.ai/free/en/papers/ding-2023-enhancing-chat-language-models-scaling/). arXiv preprint arXiv:2305.14233.
11. Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, et al. 2024. [Loramoe: Alleviating world knowledge forgetting in large language models via moe-style plugin](https://md.research.elamur.ai/free/en/papers/dou-2024-loramoe-alleviating-world-knowledge-forgetting/). In Proceedings of the 62nd Annual Meeting of the Association for Computatio…
12. Qianlong Du, Chengqing Zong, and Jiajun Zhang. 2023. Mods: Model-oriented data selection for instruction tuning. arXiv preprint arXiv:2311.15653. — [arXiv:2311.15653](https://arxiv.org/abs/2311.15653) [find](https://scholar.google.com/scholar?q=Qianlong+Du%2C+Chengqing+Zong%2C+and+Jiajun+Zhang%2E+2023%2E+Mods%3A+Model%2Doriented+data+selection+for+instruction+tuning%2E+arXiv+preprint+arXiv%3A2311%2E15653%2E+%E2%80%94+%5BarXiv%3A2311%2E15653%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2311%2E15653%29)
13. Wenchao Du and Alan W Black. 2019. Boosting dialog response generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Wenchao+Du+and+Alan+W+Black%2E+2019%2E+Boosting+dialog+response+generation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
14. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
15. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. [find](https://scholar.google.com/scholar?q=Yann+Dubois%2C+Bal%C3%A1zs+Galambosi%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2024%2E+Length%2Dcontrolled+alpacaeval%3A+A+simple+way+to+debias+automatic+evaluators%2E+arXiv+preprint+arXiv%3A2404%2E04475%2E)
16. Yuval Eldar, Michael Lindenbaum, Moshe Porat, and Yehoshua Y Zeevi. 1997. The farthest point strategy for progressive image sampling. IEEE transactions on image processing, 6(9):1305–1315. [find](https://scholar.google.com/scholar?q=Yuval+Eldar%2C+Michael+Lindenbaum%2C+Moshe+Porat%2C+and+Yehoshua+Y+Zeevi%2E+1997%2E+The+farthest+point+strategy+for+progressive+image+sampling%2E+IEEE+transactions+on+image+processing%2C+6%289%29%3A1305%E2%80%931315%2E)
17. Reza Zanjirani Farahani and Masoud Hekmatfar. 2009. Facility location: concepts, models, algorithms and case studies. Springer Science & Business Media. [find](https://scholar.google.com/scholar?q=Reza+Zanjirani+Farahani+and+Masoud+Hekmatfar%2E+2009%2E+Facility+location%3A+concepts%2C+models%2C+algorithms+and+case+studies%2E+Springer+Science+%26+Business+Media%2E)
18. William J Cook, William H Cunningham, William R Pulleyblank, and Alexander Schrijver. 1994. Combinatorial optimization. Unpublished manuscript, 10:75–93. [find](https://scholar.google.com/scholar?q=William+J+Cook%2C+William+H+Cunningham%2C+William+R+Pulleyblank%2C+and+Alexander+Schrijver%2E+1994%2E+Combinatorial+optimization%2E+Unpublished+manuscript%2C+10%3A75%E2%80%9393%2E)
19. Sándor P Fekete and Henk Meijer. 2004. Maximum dispersion and geometric maximum weight cliques. Algorithmica, 38:501–511. [find](https://scholar.google.com/scholar?q=S%C3%A1ndor+P+Fekete+and+Henk+Meijer%2E+2004%2E+Maximum+dispersion+and+geometric+maximum+weight+cliques%2E+Algorithmica%2C+38%3A501%E2%80%93511%2E)
20. Vu Minh Hoang Dang and Rakesh M Verma. 2024. Data quality in nlp: Metrics and a comprehensive taxonomy. In International Symposium on Intelligent Data Analysis, pages 217–229. Springer. [find](https://scholar.google.com/scholar?q=Vu+Minh+Hoang+Dang+and+Rakesh+M+Verma%2E+2024%2E+Data+quality+in+nlp%3A+Metrics+and+a+comprehensive+taxonomy%2E+In+International+Symposium+on+Intelligent+Data+Analysis%2C+pages+217%E2%80%93229%2E+Springer%2E)
21. Yuan Ge, Yilun Liu, Chi Hu, Weibin Meng, Shimin Tao, Xiaofeng Zhao, Hongxia Ma, Li Zhang, Boxing Chen, Hao Yang, et al. 2024. Clustering and ranking: Diversity-preserved instruction selection through expert-aligned quality estimation. arXiv preprint arXiv:2402.18191. [find](https://scholar.google.com/scholar?q=Yuan+Ge%2C+Yilun+Liu%2C+Chi+Hu%2C+Weibin+Meng%2C+Shimin+Tao%2C+Xiaofeng+Zhao%2C+Hongxia+Ma%2C+Li+Zhang%2C+Boxing+Chen%2C+Hao+Yang%2C+et+al%2E+2024%2E+Clustering+and+ranking%3A+Diversity%2Dpreserved+instruction+selection+through+)
22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
23. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. [Billion-scale similarity search with GPUs](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). IEEE Transactions on Big Data, 7(3):535–547.
24. Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. 2023. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. arXiv preprint arXiv:2311.00288. [find](https://scholar.google.com/scholar?q=Po%2DNien+Kung%2C+Fan+Yin%2C+Di+Wu%2C+Kai%2DWei+Chang%2C+and+Nanyun+Peng%2E+2023%2E+Active+instruction+tuning%3A+Improving+cross%2Dtask+generalization+by+training+on+prompt+sensitive+tasks%2E+arXiv+preprint+arXiv%3A2311%2E0028)
25. Yi-An Lai, Xuan Zhu, Yi Zhang, and Mona Diab. 2020. Diversity, density, and homogeneity: Quantitative characteristic metrics for text collections. arXiv preprint arXiv:2003.08529. [find](https://scholar.google.com/scholar?q=Yi%2DAn+Lai%2C+Xuan+Zhu%2C+Yi+Zhang%2C+and+Mona+Diab%2E+2020%2E+Diversity%2C+density%2C+and+homogeneity%3A+Quantitative+characteristic+metrics+for+text+collections%2E+arXiv+preprint+arXiv%3A2003%2E08529%2E)
26. Yunfan Shao, Linyang Li, Zhaoye Fei, Hang Yan, Dahua Lin, and Xipeng Qiu. 2024. Balanced data sampling for language model training with clustering. arXiv preprint arXiv:2402.14526. [find](https://scholar.google.com/scholar?q=Yunfan+Shao%2C+Linyang+Li%2C+Zhaoye+Fei%2C+Hang+Yan%2C+Dahua+Lin%2C+and+Xipeng+Qiu%2E+2024%2E+Balanced+data+sampling+for+language+model+training+with+clustering%2E+arXiv+preprint+arXiv%3A2402%2E14526%2E)
27. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [find](https://scholar.google.com/scholar?q=Xuechen+Li%2C+Tianyi+Zhang%2C+Yann+Dubois%2C+Rohan+Taori%2C+Ishaan+Gulrajani%2C+Carlos+Guestrin%2C+Percy+Liang%2C+and+Tatsunori+B+Hashimoto%2E+2023%2E+Alpacaeval%3A+An+automatic+evaluator+of+instruction%2Dfollowing+models%2E)
28. Jielin Song, Siyu Liu, Bin Zhu, and Yanghui Rao. 2024. Iterselecttune: An iterative training framework for efficient instruction-tuning data selection. arXiv preprint arXiv:2410.13464. — [arXiv:2410.13464](https://arxiv.org/abs/2410.13464) [find](https://scholar.google.com/scholar?q=Jielin+Song%2C+Siyu+Liu%2C+Bin+Zhu%2C+and+Yanghui+Rao%2E+2024%2E+Iterselecttune%3A+An+iterative+training+framework+for+efficient+instruction%2Dtuning+data+selection%2E+arXiv+preprint+arXiv%3A2410%2E13464%2E+%E2%80%94+%5BarXiv%3A2410%2E1)
29. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023. [What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/). arXiv preprint arXiv:2312.15685.
30. Katherine Stasaski and Marti A Hearst. 2022. Semantic diversity in dialogue with natural language inference. arXiv preprint arXiv:2205.01497. [find](https://scholar.google.com/scholar?q=Katherine+Stasaski+and+Marti+A+Hearst%2E+2022%2E+Semantic+diversity+in+dialogue+with+natural+language+inference%2E+arXiv+preprint+arXiv%3A2205%2E01497%2E)
31. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. 2023. [The flan collection: Designing data and methods for effective instruction tuning](https://md.research.elamur.ai/free/en/papers/longpre-2023-flan-collection-designing-data-methods/). In International Conference on Machine Learning, pages 22631–22648. PMLR.
32. James MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, volume 5, pages 281–298. University of California press. [find](https://scholar.google.com/scholar?q=James+MacQueen%2E+1967%2E+Some+methods+for+classification+and+analysis+of+multivariate+observations%2E+In+Proceedings+of+the+Fifth+Berkeley+Symposium+on+Mathematical+Statistics+and+Probability%2C+Volume+1%3A+St)
33. David Malvern, Brian Richards, Ngoni Chipere, and Pilar Durán. 2004. Lexical diversity and language development. Springer. [find](https://scholar.google.com/scholar?q=David+Malvern%2C+Brian+Richards%2C+Ngoni+Chipere%2C+and+Pilar+Dur%C3%A1n%2E+2004%2E+Lexical+diversity+and+language+development%2E+Springer%2E)
34. Amey P Pasarkar and Adji Bousso Dieng. 2023. Cousins of the vendi score: A family of similarity-based diversity metrics for science and machine learning. arXiv preprint arXiv:2310.12952. — [arXiv:2310.12952](https://arxiv.org/abs/2310.12952) [find](https://scholar.google.com/scholar?q=Amey+P+Pasarkar+and+Adji+Bousso+Dieng%2E+2023%2E+Cousins+of+the+vendi+score%3A+A+family+of+similarity%2Dbased+diversity+metrics+for+science+and+machine+learning%2E+arXiv+preprint+arXiv%3A2310%2E12952%2E+%E2%80%94+%5BarXiv%3A2310)
35. Yulei Qin, Yuncheng Yang, Pengcheng Guo, Gang Li, Hang Shao, Yuchen Shi, Zihan Xu, Yun Gu, Ke Li, and Xing Sun. 2024. Unleashing the power of data tsunami: A comprehensive survey on data assessment and selection for instruction tuning of language models. arXiv preprint arXiv:2408.02085. — [arXiv:2408.02085](https://arxiv.org/abs/2408.02085) [find](https://scholar.google.com/scholar?q=Yulei+Qin%2C+Yuncheng+Yang%2C+Pengcheng+Guo%2C+Gang+Li%2C+Hang+Shao%2C+Yuchen+Shi%2C+Zihan+Xu%2C+Yun+Gu%2C+Ke+Li%2C+and+Xing+Sun%2E+2024%2E+Unleashing+the+power+of+data+tsunami%3A+A+comprehensive+survey+on+data+assessment+an)
36. Sekharipuram S Ravi, Daniel J Rosenkrantz, and Giri Kumar Tayi. 1994. Heuristic and special case algorithms for dispersion problems. Operations research, 42(2):299–310. [find](https://scholar.google.com/scholar?q=Sekharipuram+S+Ravi%2C+Daniel+J+Rosenkrantz%2C+and+Giri+Kumar+Tayi%2E+1994%2E+Heuristic+and+special+case+algorithms+for+dispersion+problems%2E+Operations+research%2C+42%282%29%3A299%E2%80%93310%2E)
37. Brian Richards. 1987. Type/token ratios: What do they really tell us? Journal of child language, 14(2):201– 209. [find](https://scholar.google.com/scholar?q=Brian+Richards%2E+1987%2E+Type%2Ftoken+ratios%3A+What+do+they+really+tell+us%3F+Journal+of+child+language%2C+14%282%29%3A201%E2%80%93+209%2E)
38. Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, et al. 2022. [Multitask prompted training enables zero-shot task generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/). In International Conference on Learning Representations.
39. Ozan Sener and Silvio Savarese. 2017. Active learning for convolutional neural networks: A core-set approach. arXiv preprint arXiv:1708.00489. [find](https://scholar.google.com/scholar?q=Ozan+Sener+and+Silvio+Savarese%2E+2017%2E+Active+learning+for+convolutional+neural+networks%3A+A+core%2Dset+approach%2E+arXiv+preprint+arXiv%3A1708%2E00489%2E)
40. Katherine Stasaski, Grace Hui Yang, and Marti A Hearst. 2020. More diverse dialogue datasets via diversity-informed data collection. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4958–4968. [find](https://scholar.google.com/scholar?q=Katherine+Stasaski%2C+Grace+Hui+Yang%2C+and+Marti+A+Hearst%2E+2020%2E+More+diverse+dialogue+datasets+via+diversity%2Dinformed+data+collection%2E+In+Proceedings+of+the+58th+annual+meeting+of+the+association+for+co)
41. Peng Sun, Bei Shi, Daiwei Yu, and Tao Lin. 2024. On the diversity and realism of distilled dataset: An efficient dataset distillation paradigm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9390–9399. [find](https://scholar.google.com/scholar?q=Peng+Sun%2C+Bei+Shi%2C+Daiwei+Yu%2C+and+Tao+Lin%2E+2024%2E+On+the+diversity+and+realism+of+distilled+dataset%3A+An+efficient+dataset+distillation+paradigm%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vi)
42. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
43. Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu. 2024a. A survey on data selection for llm instruction tuning. arXiv preprint arXiv:2402.05123. — [arXiv:2402.05123](https://arxiv.org/abs/2402.05123) [find](https://scholar.google.com/scholar?q=Jiahao+Wang%2C+Bolin+Zhang%2C+Qianlong+Du%2C+Jiajun+Zhang%2C+and+Dianhui+Chu%2E+2024a%2E+A+survey+on+data+selection+for+llm+instruction+tuning%2E+arXiv+preprint+arXiv%3A2402%2E05123%2E+%E2%80%94+%5BarXiv%3A2402%2E05123%5D%28https%3A%2F%2Farxiv%2E)
44. Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. 2024b. Diversity measurement and subset selection for instruction tuning datasets. arXiv preprint arXiv:2402.02318. [find](https://scholar.google.com/scholar?q=Peiqi+Wang%2C+Yikang+Shen%2C+Zhen+Guo%2C+Matthew+Stallone%2C+Yoon+Kim%2C+Polina+Golland%2C+and+Rameswar+Panda%2E+2024b%2E+Diversity+measurement+and+subset+selection+for+instruction+tuning+datasets%2E+arXiv+preprint+arX)
45. Shengguang Wu, Keming Lu, Benfeng Xu, Junyang Lin, Qi Su, and Chang Zhou. 2023. Self-evolved diverse data sampling for efficient instruction tuning. arXiv preprint arXiv:2311.08182. — [arXiv:2311.08182](https://arxiv.org/abs/2311.08182) [find](https://scholar.google.com/scholar?q=Shengguang+Wu%2C+Keming+Lu%2C+Benfeng+Xu%2C+Junyang+Lin%2C+Qi+Su%2C+and+Chang+Zhou%2E+2023%2E+Self%2Devolved+diverse+data+sampling+for+efficient+instruction+tuning%2E+arXiv+preprint+arXiv%3A2311%2E08182%2E+%E2%80%94+%5BarXiv%3A2311%2E0818)
46. Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, and Junyang Lin. 2024. Rethinking data selection at scale: Random selection is almost all you need. arXiv preprint arXiv:2410.09335. [find](https://scholar.google.com/scholar?q=Tingyu+Xia%2C+Bowen+Yu%2C+Kai+Dang%2C+An+Yang%2C+Yuan+Wu%2C+Yuan+Tian%2C+Yi+Chang%2C+and+Junyang+Lin%2E+2024%2E+Rethinking+data+selection+at+scale%3A+Random+selection+is+almost+all+you+need%2E+arXiv+preprint+arXiv%3A2410%2E093)
47. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. 2024. [Wizardlm: Empowering large pre-trained language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). In The Twelfth International Conference on Learning Representations.
48. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+et+al%2E+2024%2E+Qwen2%2E+5+technical+report%2E+arXiv+preprint+arXiv%3A2412%2E15115%2E)
49. Yuming Yang, Wantong Zhao, Caishuang Huang, Junjie Ye, Xiao Wang, Huiyuan Zheng, Yang Nan, Yuran Wang, Xueying Xu, Kaixin Huang, Yunke Zhang, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. [Beyond boundaries: Learning a universal entity taxonomy across datasets and languages for open named entity recog…](https://md.research.elamur.ai/free/en/papers/yang-2025-beyond-boundaries-learning-universal-entity/)
50. Junjie Ye, Yuming Yang, Qi Zhang, Tao Gui, Xuanjing Huang, Peng Wang, Zhongchao Shi, and Jianping Fan. 2024. Empirical insights on fine-tuning large language models for question-answering. CoRR, abs/2409.15825. — [arXiv:2409.15825](https://arxiv.org/abs/2409.15825) [find](https://scholar.google.com/scholar?q=Junjie+Ye%2C+Yuming+Yang%2C+Qi+Zhang%2C+Tao+Gui%2C+Xuanjing+Huang%2C+Peng+Wang%2C+Zhongchao+Shi%2C+and+Jianping+Fan%2E+2024%2E+Empirical+insights+on+fine%2Dtuning+large+language+models+for+question%2Danswering%2E+CoRR%2C+abs%2F2)
51. Yu Yu, Shahram Khadivi, and Jia Xu. 2022. Can data diversity enhance learning generalization? In Proceedings of the 29th international conference on computational linguistics, pages 4933–4945. [find](https://scholar.google.com/scholar?q=Yu+Yu%2C+Shahram+Khadivi%2C+and+Jia+Xu%2E+2022%2E+Can+data+diversity+enhance+learning+generalization%3F+In+Proceedings+of+the+29th+international+conference+on+computational+linguistics%2C+pages+4933%E2%80%934945%2E)
52. Jerrold H Zar. 2005. Spearman rank correlation. Encyclopedia of biostatistics, 7. [find](https://scholar.google.com/scholar?q=Jerrold+H+Zar%2E+2005%2E+Spearman+rank+correlation%2E+Encyclopedia+of+biostatistics%2C+7%2E)
53. Dylan Zhang, Justin Wang, and Francois Charton. 2024a. Instruction diversity drives generalization to unseen tasks. arXiv preprint arXiv:2402.10891. — [arXiv:2402.10891](https://arxiv.org/abs/2402.10891) [find](https://scholar.google.com/scholar?q=Dylan+Zhang%2C+Justin+Wang%2C+and+Francois+Charton%2E+2024a%2E+Instruction+diversity+drives+generalization+to+unseen+tasks%2E+arXiv+preprint+arXiv%3A2402%2E10891%2E+%E2%80%94+%5BarXiv%3A2402%2E10891%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2402%2E108)
54. Dylan Zhang, Justin Wang, and Francois Charton. 2024b. Only-IF: Revealing the decisive effect of instruction diversity on generalization. arXiv preprint arXiv:2410.04717. — [arXiv:2410.04717](https://arxiv.org/abs/2410.04717) [find](https://scholar.google.com/scholar?q=Dylan+Zhang%2C+Justin+Wang%2C+and+Francois+Charton%2E+2024b%2E+Only%2DIF%3A+Revealing+the+decisive+effect+of+instruction+diversity+on+generalization%2E+arXiv+preprint+arXiv%3A2410%2E04717%2E+%E2%80%94+%5BarXiv%3A2410%2E04717%5D%28https%3A%2F%2F)
55. Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023. [Instruction tuning for large language models: A survey](https://md.research.elamur.ai/free/en/papers/zhang-2023-instruction-tuning-large-language-models/). arXiv preprint arXiv:2308.10792.
56. Dorothy Zhao, Jerone TA Andrews, AI Sony, Tokyo Orestis Papakyriakopoulos, and Alice Xiang. 2024a. Measuring diversity in datasets. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dorothy+Zhao%2C+Jerone+TA+Andrews%2C+AI+Sony%2C+Tokyo+Orestis+Papakyriakopoulos%2C+and+Alice+Xiang%2E+2024a%2E+Measuring+diversity+in+datasets%2E+In+International+Conference+on+Learning+Representations%2E)
57. Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024b. Long is more for alignment: A simple but tough-to-beat baseline for instruction fine-tuning. arXiv preprint arXiv:2402.04833. — [arXiv:2402.04833](https://arxiv.org/abs/2402.04833) [find](https://scholar.google.com/scholar?q=Hao+Zhao%2C+Maksym+Andriushchenko%2C+Francesco+Croce%2C+and+Nicolas+Flammarion%2E+2024b%2E+Long+is+more+for+alignment%3A+A+simple+but+tough%2Dto%2Dbeat+baseline+for+instruction+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2402%2E)
58. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). Advances in Neural Information Processing Systems, 36:46595–46623.
59. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2024. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srinivasan+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2024%2E+Lima%3A+Less+is+more+for+alignment%2E+Advances+in+Neural+Information+Proc)

*Updated 31 October 2026*