---
title: "Automatic Instruction Evolving for Large Language Models (2024)"
description: "This paper presents Auto Evol-Instruct, which removes that manual step. 37 references, cited by 23 papers here."
authors: ["Weihao Zeng", "Can Xu", "Yingxiu Zhao", "Jian-Guang Lou", "Weizhu Chen"]
year: 2024
venue: "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2024.emnlp-main.397/"
doi: "10.18653/v1/2024.emnlp-main.397"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zeng-2024-automatic-instruction-evolving-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zeng-2024-automatic-instruction-evolving-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / Automatic Instruction Evolving for Large Language Models

# Automatic Instruction Evolving for Large Language Models

**Authors:** [Weihao Zeng](https://md.research.elamur.ai/free/en/authors/weihao-zeng/), [Can Xu](https://md.research.elamur.ai/free/en/authors/can-xu/), [Yingxiu Zhao](https://md.research.elamur.ai/free/en/authors/yingxiu-zhao/), [Jian-Guang Lou](https://md.research.elamur.ai/free/en/authors/jian-guang-lou/) and [Weizhu Chen](https://md.research.elamur.ai/free/en/authors/weizhu-chen/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2024](https://md.research.elamur.ai/free/en/venues/emnlp/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.emnlp-main.397/) · [PDF](https://aclanthology.org/2024.emnlp-main.397.pdf) · [DOI 10.18653/v1/2024.emnlp-main.397](https://doi.org/10.18653/v1/2024.emnlp-main.397) · [OpenAlex](https://openalex.org/W4404782750)

**Citations:** 5 citations (OpenAlex, 2026-10-14) · [23](https://md.research.elamur.ai/free/en/papers/zeng-2024-automatic-instruction-evolving-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Evol-Instruct improves instruction-tuning data by having an LLM rewrite instructions into harder, more varied versions. However, its rewriting rules are hand-crafted by experts and must be redesigned for each new domain. This paper presents Auto Evol-Instruct, which removes that manual step.

The process starts from a generic prompt. In it, the evol LLM brainstorms ways to complicate an instruction, plans and applies some of them, and then reviews the result. An optimizer LLM then examines the multi-round rewriting trajectories on small sampled batches, about 2,000 seed items in total. It diagnoses failures, such as complexity not increasing, unsolvable tasks, or the model answering rather than rewriting. It then revises the evolving prompt to fix them. Several candidate revisions are produced in parallel, and the one with the lowest failure rate is kept. The final method is applied to the whole dataset.

Across chat, math and code, models tuned on the automatically evolved data beat those tuned on data evolved with human-designed methods. The extra cost is only a few thousand API calls. This makes data evolution cheap to transfer to new tasks.

## Key points
- The framework alternates two stages, Evol Trajectory Analysis and Evolving Method Optimization. It mainly uses GPT-3.5-Turbo and GPT-4 as the evol and optimizer LLMs.
- Mixtral-8x7B tuned on 10K evolved ShareGPT samples reaches 8.09 on MT-Bench and 91.4 on AlpacaEval. This exceeds GPT-3.5-Turbo and WizardLM-70B.
- Mixtral-8x7B tuned on 7K evolved GSM8K items scores 82.49 on GSM8K. This exceeds WizardMath-70B and MetaMath-70B.
- DeepSeek-Coder-Base-33B tuned on 20K evolved Code Alpaca samples reaches 77.4 on HumanEval. This beats WizardCoder-34B.
- The authors list several limitations: no evaluation on MMLU or TruthfulQA, a limited set of base models (Mistral, Mixtral-8x7B, CodeLlama-13B-Python, DeepSeek-Coder-Base-33B), and fairly simple optimization prompts.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou and Weizhu Chen. 2024. Automatic Instruction Evolving for Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{zeng2024automatic,
  title = {{Automatic Instruction Evolving for Large Language Models}},
  author = {Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou and Weizhu Chen},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.397},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.397}
}
```

- [← previous ArMeme: Propagandistic Content in Arabic Memes](https://md.research.elamur.ai/free/en/papers/alam-2024-armeme-propagandistic-content-arabic-memes/)
- [next → AutoPSV: Automated Process-Supervised Verifier](https://md.research.elamur.ai/free/en/papers/lu-2024-autopsv-automated-process-supervised-verifier/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

37 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
2. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. [find](https://scholar.google.com/scholar?q=Sahil+Chaudhary%2E+2023%2E+Code+alpaca%3A+An+instruction%2Dfollowing+llama+model+for+code+generation%2E)
3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
4. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See [https://vicuna.lmsys](https://vicuna.lmsys). org (accessed 14 April 2023). [↗](https://vicuna.lmsys)
5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? try arc, the ai2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
7. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
8. Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. [Alpacafarm: A simulation framework for methods that learn from human feedback](https://md.research.elamur.ai/free/en/papers/dubois-2023-alpacafarm-simulation-framework-methods-that/). Preprint, arXiv:2305.14387.
9. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y+Wu%2C+YK+Li%2C+et+al%2E+2024%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise+of+code)
10. Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. 2023. Connecting large language models with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint arXiv:2309.08532. [find](https://scholar.google.com/scholar?q=Qingyan+Guo%2C+Rui+Wang%2C+Junliang+Guo%2C+Bei+Li%2C+Kaitao+Song%2C+Xu+Tan%2C+Guoqing+Liu%2C+Jiang+Bian%2C+and+Yujiu+Yang%2E+2023%2E+Connecting+large+language+models+with+evolutionary+algorithms+yields+powerful+prompt+op)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
12. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+et+al%2E+2023%2E+Mist)
13. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088. [find](https://scholar.google.com/scholar?q=Albert+Q+Jiang%2C+Alexandre+Sablayrolles%2C+Antoine+Roux%2C+Arthur+Mensch%2C+Blanche+Savary%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Emma+Bou+Hanna%2C+Florian+Bressand%2C+et+al%2E+2024%2E+Mixtral+o)
14. Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, et al. 2023. [Openassistant conversations–democratizing large language model alignment](https://md.research.elamur.ai/free/en/papers/kopf-2023-openassistant-conversations-democratizing-large-language/). arXiv preprint arXiv:2304.07327.
15. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. [https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval). [↗](https://github.com/tatsu-lab/alpaca_eval)
16. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. [TruthfulQA: Measuring how models mimic human falsehoods](https://md.research.elamur.ai/free/en/papers/lin-2022-truthfulqa-measuring-how-models-mimic/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
17. Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang. 2023a. Tinygsm: achieving\> 80% on gsm8k with small language models. arXiv preprint arXiv:2312.09241. [find](https://scholar.google.com/scholar?q=Bingbin+Liu%2C+Sebastien+Bubeck%2C+Ronen+Eldan%2C+Janardhan+Kulkarni%2C+Yuanzhi+Li%2C+Anh+Nguyen%2C+Rachel+Ward%2C+and+Yi+Zhang%2E+2023a%2E+Tinygsm%3A+achieving%5C%3E+80%25+on+gsm8k+with+small+language+models%2E+arXiv+preprint+a)
18. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2023b. [What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning](https://md.research.elamur.ai/free/en/papers/liu-2023-makes-good-data-alignment-comprehensive/). Preprint, arXiv:2312.15685.
19. Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. \# instag: Instruction tagging for analyzing supervised fine-tuning of large language models. arXiv e-prints, pages arXiv–2308. [find](https://scholar.google.com/scholar?q=Keming+Lu%2C+Hongyi+Yuan%2C+Zheng+Yuan%2C+Runji+Lin%2C+Junyang+Lin%2C+Chuanqi+Tan%2C+Chang+Zhou%2C+and+Jingren+Zhou%2E+2023%2E+%5C%23+instag%3A+Instruction+tagging+for+analyzing+supervised+fine%2Dtuning+of+large+language+model)
20. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023a. [Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct](https://md.research.elamur.ai/free/en/papers/luo-2023-wizardmath-empowering-mathematical-reasoning-large/). arXiv preprint arXiv:2308.09583.
21. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023b. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2023b%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+)
22. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv:2303.08774.
23. Preprint, Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35:27730–27744…
24. Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021. [{ZeRO-Offload}: Democratizing {Billion-Scale} model training](https://md.research.elamur.ai/free/en/papers/ren-2021-zero-offload-democratizing-billion-scale/). In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pages 551–564.
25. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
26. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. [Judging llm-as-a-judge with mt-bench and chatbot arena](https://md.research.elamur.ai/free/en/papers/zheng-2023-judging-llm-judge-mt-bench/). arXiv preprint arXiv:2306.05685.
27. Mirac Suzgun and Adam Tauman Kalai. 2024. [Meta-prompting: Enhancing language models with task-agnostic scaffolding](https://md.research.elamur.ai/free/en/papers/suzgun-2024-meta-prompting-enhancing-language-models/). arXiv preprint arXiv:2401.12954.
28. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. 2023. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. [find](https://scholar.google.com/scholar?q=Chunting+Zhou%2C+Pengfei+Liu%2C+Puxin+Xu%2C+Srini+Iyer%2C+Jiao+Sun%2C+Yuning+Mao%2C+Xuezhe+Ma%2C+Avia+Efrat%2C+Ping+Yu%2C+Lili+Yu%2C+et+al%2E+2023%2E+Lima%3A+Less+is+more+for+alignment%2E+arXiv+preprint+arXiv%3A2305%2E11206%2E)
29. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
30. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971.
31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288.
32. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
33. Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. [Wizardlm: Empowering large language models to follow complex instructions](https://md.research.elamur.ai/free/en/papers/xu-2023-wizardlm-empowering-large-pre-trained/). arXiv preprint arXiv:2304.12244.
34. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2023. [Large language models as optimizers](https://md.research.elamur.ai/free/en/papers/yang-2023-large-language-models-optimizers/). arXiv preprint arXiv:2309.03409.
35. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. [Metamath: Bootstrap your own mathematical questions for large language models](https://md.research.elamur.ai/free/en/papers/yu-2023-metamath-bootstrap-your-own-mathematical/). arXiv preprint arXiv:2309.12284.
36. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830. [find](https://scholar.google.com/scholar?q=Rowan+Zellers%2C+Ari+Holtzman%2C+Yonatan+Bisk%2C+Ali+Farhadi%2C+and+Yejin+Choi%2E+2019%2E+Hellaswag%3A+Can+a+machine+really+finish+your+sentence%3F+arXiv+preprint+arXiv%3A1905%2E07830%2E)
37. Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L Zhang. 2023. A preliminary study of the intrinsic relationship between complexity and alignment. arXiv preprint arXiv:2308.05696. — [arXiv:2308.05696](https://arxiv.org/abs/2308.05696) [find](https://scholar.google.com/scholar?q=Yingxiu+Zhao%2C+Bowen+Yu%2C+Binyuan+Hui%2C+Haiyang+Yu%2C+Fei+Huang%2C+Yongbin+Li%2C+and+Nevin+L+Zhang%2E+2023%2E+A+preliminary+study+of+the+intrinsic+relationship+between+complexity+and+alignment%2E+arXiv+preprint+arXi)

*Updated 30 October 2026*