---
title: "CodeFlowBench — paper by Wang et al. (2026)"
description: "The authors call this incremental, reuse-driven process \"codeflow\" and present CodeFlowBench as the first benchmark for… 36 references."
authors: ["Sizhe Wang", "Zhengren Wang", "Dongsheng Ma", "Yongan Yu", "Rui Ling", "Zhiyu li", "Feiyu Xiong", "Wentao Zhang"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.201/"
doi: "10.18653/v1/2026.acl-long.201"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-codeflowbench-multi-turn-iterative-benchmark/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-codeflowbench-multi-turn-iterative-benchmark/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/) / CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation

# CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation

**Authors:** [Sizhe Wang](https://md.research.elamur.ai/free/en/authors/sizhe-wang/), [Zhengren Wang](https://md.research.elamur.ai/free/en/authors/zhengren-wang/), Dongsheng Ma, Yongan Yu, [Rui Ling](https://md.research.elamur.ai/free/en/authors/rui-ling/), [Zhiyu li](https://md.research.elamur.ai/free/en/authors/zhiyu-li/), [Feiyu Xiong](https://md.research.elamur.ai/free/en/authors/feiyu-xiong/) and [Wentao Zhang](https://md.research.elamur.ai/free/en/authors/wentao-zhang/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.201/) · [PDF](https://aclanthology.org/2026.acl-long.201.pdf) · [DOI 10.18653/v1/2026.acl-long.201](https://doi.org/10.18653/v1/2026.acl-long.201) · [OpenAlex](https://openalex.org/W4417525503)

**Citations:** 1 citations (OpenAlex, 2026-10-14)

## Summary
Most code-generation benchmarks test LLMs in a single turn, but real software is built step by step from modular functions that reuse earlier ones. The authors call this incremental, reuse-driven process "codeflow" and present CodeFlowBench as the first benchmark for evaluating it.

An automated pipeline based on function dependency analysis breaks monolithic solutions into ordered subproblems, each with a verified solution and unit tests. The benchmark has two parts:
- **CodeFlowBench-Comp** draws on Codeforces problems and is refreshed periodically to limit contamination.
- **CodeFlowBench-Repo** draws on GitHub code collected through DomainEval.

The evaluation framework compares multi-turn and single-turn generation. It also adds structural metrics computed from dependency trees.

Across 19 LLMs, accuracy drops sharply in the multi-turn setting, by up to 50%. Scores also decline as dependency structures grow more complex. This shows that even strong reasoning models struggle to reuse earlier code and keep the overall goal in view, and it gives researchers an updatable testbed for workflows closer to real development.

## Key points
- **CodeFlowBench-Comp** contains over 5,000 Codeforces problems, maintained by an automated update pipeline. **CodeFlowBench-Repo** is built from real GitHub repositories via DomainEval.
- **New metrics:** Average Pass Depth (APD@k) and Dependency Structure Complexity (DSC), both derived from function dependency trees.
- **Main result:** among the 19 LLMs tested, multi-turn performance falls by as much as 50% compared with single-turn. Pass@1 decreases as DSC increases.
- **Error analysis:** failures fall into three categories labeled IR, IG, and Instruction Misinterpretation (IM).
  - Comp errors are dominated by IR, while Repo shifts the balance toward IG and IM.
  - DeepSeek-V3 and o1-mini are limited mainly by IM; for o1-mini on Repo, IM accounts for 55.9% of errors.
  - For Gemini-3-Flash, IR is the main bottleneck.
- **Stated limitations:** the data preparation step is specific to its sources, and no models fine-tuned for codeflow exist yet.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 102 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu li, Feiyu Xiong and Wentao Zhang. 2026. CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026codeflowbench,
  title = {{CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation}},
  author = {Sizhe Wang, Zhengren Wang, Dongsheng Ma, Yongan Yu, Rui Ling, Zhiyu li, Feiyu Xiong and Wentao Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.201},
  url = {https://doi.org/10.18653/v1/2026.acl-long.201}
}
```

- [← previous CodeEvo: Interaction-Driven Synthesis of Code-centric Data through Hybrid and Iterative Feedback](https://md.research.elamur.ai/free/en/papers/sun-2026-codeevo-interaction-driven-synthesis-code/)
- [next → CodeMEM: AST-Guided Adaptive Memory for Repository-Level Iterative Code Generation](https://md.research.elamur.ai/free/en/papers/wang-2026-codemem-ast-guided-adaptive-memory/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/11/)

## References

36 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pekka Abrahamsson, Outi Salo, Jussi Ronkainen, and Juhani Warsta. 2017. Agile software development methods: Review and analysis. arXiv preprint arXiv:1709.08439. — [arXiv:1709.08439](https://arxiv.org/abs/1709.08439) [find](https://scholar.google.com/scholar?q=Pekka+Abrahamsson%2C+Outi+Salo%2C+Jussi+Ronkainen%2C+and+Juhani+Warsta%2E+2017%2E+Agile+software+development+methods%3A+Review+and+analysis%2E+arXiv+preprint+arXiv%3A1709%2E08439%2E+%E2%80%94+%5BarXiv%3A1709%2E08439%5D%28https%3A%2F%2Farxiv%2Eorg)
2. Anthropic. 2025. Claude-4 system card. System card, Anthropic. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025%2E+Claude%2D4+system+card%2E+System+card%2C+Anthropic%2E)
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
4. Adam Boduch. 2019. React material-ui cookbook: build captivating user experiences using react and material-ui. Packt Publishing Ltd. [find](https://scholar.google.com/scholar?q=Adam+Boduch%2E+2019%2E+React+material%2Dui+cookbook%3A+build+captivating+user+experiences+using+react+and+material%2Dui%2E+Packt+Publishing+Ltd%2E)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
6. W Alec Cram. 2019. Agile development in practice: Lessons from the trenches. Information Systems Management, 36(1):2–14. [find](https://scholar.google.com/scholar?q=W+Alec+Cram%2E+2019%2E+Agile+development+in+practice%3A+Lessons+from+the+trenches%2E+Information+Systems+Management%2C+36%281%29%3A2%E2%80%9314%2E)
7. Daniel Feitosa, Apostolos Ampatzoglou, Antonios Gko-rtzis, Stamatia Bibi, and Alexander Chatzigeorgiou. 2020. Code reuse in practice: Benefiting or harming technical debt. Journal of Systems and Software, 167:110618. [find](https://scholar.google.com/scholar?q=Daniel+Feitosa%2C+Apostolos+Ampatzoglou%2C+Antonios+Gko%2Drtzis%2C+Stamatia+Bibi%2C+and+Alexander+Chatzigeorgiou%2E+2020%2E+Code+reuse+in+practice%3A+Benefiting+or+harming+technical+debt%2E+Journal+of+Systems+and+Softw)
8. Google. 2025. Gemini-3-flash system card. System card, Google. [find](https://scholar.google.com/scholar?q=Google%2E+2025%2E+Gemini%2D3%2Dflash+system+card%2E+System+card%2C+Google%2E)
9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv preprint arXiv:2407.21783.
10. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. [Cruxeval: A benchmark for code reasoning, understanding and execution](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/). arXiv preprint arXiv:2401.03065. llm-based agents for software engineering: A survey of current, challenges and future. arXiv…
11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
12. Craig Larman. 2004. Agile and iterative development: a manager’s guide. Addison-Wesley Professional. [find](https://scholar.google.com/scholar?q=Craig+Larman%2E+2004%2E+Agile+and+iterative+development%3A+a+manager%E2%80%99s+guide%2E+Addison%2DWesley+Professional%2E)
13. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Yu+Wu%2C+YK+Li%2C+and+1+others%2E+2024%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise)
14. Stefan Haefliger, Georg Von Krogh, and Sebastian Spaeth. 2008. Code reuse in open source software. Management science, 54(1):180–193. [find](https://scholar.google.com/scholar?q=Stefan+Haefliger%2C+Georg+Von+Krogh%2C+and+Sebastian+Spaeth%2E+2008%2E+Code+reuse+in+open+source+software%2E+Management+science%2C+54%281%29%3A180%E2%80%93193%2E)
15. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and 1 others. 2021. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). arXiv preprint arXiv:2105.09938.
16. Dong Huang, Yuhao Qing, Weiyi Shang, Heming Cui, and Jie Zhang. 2024. Effibench: Benchmarking the efficiency of automatically generated code. Advances in Neural Information Processing Systems, 37:11506– 11544. [find](https://scholar.google.com/scholar?q=Dong+Huang%2C+Yuhao+Qing%2C+Weiyi+Shang%2C+Heming+Cui%2C+and+Jie+Zhang%2E+2024%2E+Effibench%3A+Benchmarking+the+efficiency+of+automatically+generated+code%2E+Advances+in+Neural+Information+Processing+Systems%2C+37%3A1150)
17. Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. 2023. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010. [find](https://scholar.google.com/scholar?q=Dong+Huang%2C+Jie+M+Zhang%2C+Michael+Luck%2C+Qingwen+Bu%2C+Yuhao+Qing%2C+and+Heming+Cui%2E+2023%2E+Agentcoder%3A+Multi%2Dagent%2Dbased+code+generation+with+iterative+testing+and+optimisation%2E+arXiv+preprint+arXiv%3A2312%2E13)
18. Md Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. Mapcoder: Multi-agent code generation for competitive problem solving. arXiv preprint arXiv:2405.11403. [find](https://scholar.google.com/scholar?q=Md+Ashraful+Islam%2C+Mohammed+Eunus+Ali%2C+and+Md+Rizwan+Parvez%2E+2024%2E+Mapcoder%3A+Multi%2Dagent+code+generation+for+competitive+problem+solving%2E+arXiv+preprint+arXiv%3A2405%2E11403%2E)
19. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974.
20. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. [A survey on large language models for code generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/). arXiv preprint arXiv:2406.00515.
21. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) arXiv preprint arXiv:2310.06770.
22. Haolin Jin, Linghan Huang, Haipeng Cai, Jun Yan, Bo Li, and Huaming Chen. 2024. From llms to Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, and 1 others. 2024. Devbench: A comprehensive benchmark for software development. CoRR. [find](https://scholar.google.com/scholar?q=Haolin+Jin%2C+Linghan+Huang%2C+Haipeng+Cai%2C+Jun+Yan%2C+Bo+Li%2C+and+Huaming+Chen%2E+2024%2E+From+llms+to+Bowen+Li%2C+Wenhan+Wu%2C+Ziwei+Tang%2C+Lin+Shi%2C+John+Yang%2C+Jinyang+Li%2C+Shunyu+Yao%2C+Chen+Qian%2C+Binyuan+Hui%2C+Qichen)
23. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097.
24. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024a%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A241)
25. Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024b. Large language model-based agents for software engineering: A survey. arXiv preprint arXiv:2409.02977. — [arXiv:2409.02977](https://arxiv.org/abs/2409.02977) [find](https://scholar.google.com/scholar?q=Junwei+Liu%2C+Kaixin+Wang%2C+Yixuan+Chen%2C+Xin+Peng%2C+Zhenpeng+Chen%2C+Lingming+Zhang%2C+and+Yiling+Lou%2E+2024b%2E+Large+language+model%2Dbased+agents+for+software+engineering%3A+A+survey%2E+arXiv+preprint+arXiv%3A2409%2E02)
26. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2023%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+a)
27. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+arX)
28. OpenAI. 2025. Gpt-5 system card. System card, OpenAI. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Gpt%2D5+system+card%2E+System+card%2C+OpenAI%2E)
29. Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, and 1 others. 2025. [Codeelo: Benchmarking competition-level code generation of llms with human-comparable elo ratings](https://md.research.elamur.ai/free/en/papers/quan-2025-codeelo-benchmarking-competition-level-code/). arXiv preprint arXiv:2501.01257.
30. Zhengren Wang, Rui Ling, Chufan Wang, Yongan Yu, Zhiyu Li, Feiyu Xiong, and Wentao Zhang. 2025. Maintaincoder: Maintainable code generation under dynamic requirements. arXiv preprint arXiv:2503.24260. — [arXiv:2503.24260](https://arxiv.org/abs/2503.24260) [find](https://scholar.google.com/scholar?q=Zhengren+Wang%2C+Rui+Ling%2C+Chufan+Wang%2C+Yongan+Yu%2C+Zhiyu+Li%2C+Feiyu+Xiong%2C+and+Wentao+Zhang%2E+2025%2E+Maintaincoder%3A+Maintainable+code+generation+under+dynamic+requirements%2E+arXiv+preprint+arXiv%3A2503%2E24260%2E)
31. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Empowering code generation with oss-instruct. arXiv preprint arXiv:2312.02120. [find](https://scholar.google.com/scholar?q=Yuxiang+Wei%2C+Zhe+Wang%2C+Jiawei+Liu%2C+Yifeng+Ding%2C+and+Lingming+Zhang%2E+2023%2E+Magicoder%3A+Empowering+code+generation+with+oss%2Dinstruct%2E+arXiv+preprint+arXiv%3A2312%2E02120%2E)
32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
33. John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. [Intercode: Standardizing and benchmarking interactive coding with execution feedback](https://md.research.elamur.ai/free/en/papers/yang-2023-intercode-standardizing-benchmarking-interactive-coding/). Advances in Neural Information Processing Systems, 36:23826–23854.
34. Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. 2024. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. arXiv preprint arXiv:2412.21199. — [arXiv:2412.21199](https://arxiv.org/abs/2412.21199) [find](https://scholar.google.com/scholar?q=Zhaojian+Yu%2C+Yilun+Zhao%2C+Arman+Cohan%2C+and+Xiao%2DPing+Zhang%2E+2024%2E+Humaneval+pro+and+mbpp+pro%3A+Evaluating+large+language+models+on+self%2Dinvoking+code+generation%2E+arXiv+preprint+arXiv%3A2412%2E21199%2E+%E2%80%94+%5BarXi)
35. Qiming Zhu, Jialun Cao, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun, and Shing-Chi Cheung. 2025. Domaineval: An auto-constructed benchmark for multi-domain code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 26148–26156. [find](https://scholar.google.com/scholar?q=Qiming+Zhu%2C+Jialun+Cao%2C+Yaojie+Lu%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Le+Sun%2C+and+Shing%2DChi+Cheung%2E+2025%2E+Domaineval%3A+An+auto%2Dconstructed+benchmark+for+multi%2Ddomain+code+generation%2E+In+Proceedings+of+the+AAAI+C)
36. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others. 2024. [Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/). arXiv preprint arXiv:2406.15877.

*Updated 1 November 2026*