---
title: "ExecVerify — paper by Tang et al. (2026) — summary and ideas"
description: "It then creates and mutates inputs and filters the samples by successful execution and by difficulty. 44 references, cited by 1 paper here."
authors: ["Lingxiao Tang", "He Ye", "Zhaoyang Chu", "Muyang Ye", "Zhongxin Liu", "Xiaoxue Ren", "Lingfeng Bao"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.631/"
doi: "10.18653/v1/2026.acl-long.631"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tang-2026-execverify-white-box-rl-verifiable/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tang-2026-execverify-white-box-rl-verifiable/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/) / ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning

# ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning

**Authors:** Lingxiao Tang, He Ye, Zhaoyang Chu, [Muyang Ye](https://md.research.elamur.ai/free/en/authors/muyang-ye/), Zhongxin Liu, [Xiaoxue Ren](https://md.research.elamur.ai/free/en/authors/xiaoxue-ren/) and Lingfeng Bao · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.631/) · [PDF](https://aclanthology.org/2026.acl-long.631.pdf) · [DOI 10.18653/v1/2026.acl-long.631](https://doi.org/10.18653/v1/2026.acl-long.631) · [OpenAlex](https://openalex.org/W7166824421)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Code LLMs, especially small ones, are poor at tracing how a program actually runs. Earlier methods fine-tune on teacher-written explanations, either I/O reasoning chains or verbalized traces. Intermediate steps are never checked, so a model can learn to copy text without understanding execution, and the training data has no difficulty control. ExecVerify addresses both problems. It synthesizes Python programs by iterating over built-in types and methods, adding nested-call and control-flow constraints step by step. It then creates and mutates inputs and filters the samples by successful execution and by difficulty. Interpreter traces become checkable questions, such as which line runs next or what value and type a variable holds. After a short SFT warm-up, RL (GRPO) rewards correct answers to these questions together with correct final outputs. A second RL stage uses unit-test rewards to carry the skill over to code generation. The resulting 7B model matches 32B models such as Qwen2.5-Coder-32B-Instruct on execution reasoning. It also gains up to 5.9% pass@1 over strong post-training baselines. This shows that verifiable stepwise rewards are more effective than pure imitation.

## Key points
- **Dataset construction:** 239,992 original and 239,466 mutated instances were generated. Execution filtering kept 201,537 and 191,463 of them. Difficulty filtering then left 119,358 training examples: only items that Qwen2.5-Coder-7B-Instruct solved in at most 3 of 10 runs at temperature 1.0 were kept.
- **White-box rewards:** these cover next-statement prediction and variable value/type prediction, in addition to output (assertion) correctness. An embedding-based check found no contamination with the test sets.
- **Evaluation benchmarks:** execution reasoning was tested on CRUXEval, LiveCodeBench-Exec and REval. Code generation was tested on EvalPlus (HumanEval/MBPP), LiveCodeBench and BigCodeBench.
- **Ablation:** the average code-generation score was 50.4 with Stage I only, 53.9 with Stage II only, and 57.1 with both stages. The pipeline also brought gains for Qwen2.5-Coder-3B-Instruct and Llama3-8B-Instruct.
- **Release and limitations:** code, data and models are released at github.com/tlx000000001/ExecVerify. The work is limited to function-level Python and costs more compute than plain SFT.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 88 works · 19,432 citations
- [Zhejiang University](https://md.research.elamur.ai/free/en/universities/zhejiang-university/) — University, Hangzhou, China · 172 works · 12,988 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Lingxiao Tang, He Ye, Zhaoyang Chu, Muyang Ye, Zhongxin Liu, Xiaoxue Ren and Lingfeng Bao. 2026. ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{tang2026execverify,
  title = {{ExecVerify: White-Box RL with Verifiable Stepwise Rewards for Code Execution Reasoning}},
  author = {Lingxiao Tang, He Ye, Zhaoyang Chu, Muyang Ye, Zhongxin Liu, Xiaoxue Ren and Lingfeng Bao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.631},
  url = {https://doi.org/10.18653/v1/2026.acl-long.631}
}
```

- [← previous EVWSD-ITA at EVALITA 2026: Overview of the Enhanced Visual Word Sense Disambiguation for Italian Task](https://md.research.elamur.ai/free/en/papers/musacchio-2026-evwsd-ita-evalita-2026-overview/)
- [next → Explainable Semantic Textual Similarity via Dissimilar Span Detection](https://md.research.elamur.ai/free/en/papers/lozano-2026-explainable-semantic-textual-similarity-dissimilar/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/12/)

## References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sahil Chaudhary. 2023. Code alpaca: An instruction-following llama model for code generation. [find](https://scholar.google.com/scholar?q=Sahil+Chaudhary%2E+2023%2E+Code+alpaca%3A+An+instruction%2Dfollowing+llama+model+for+code+generation%2E)
2. Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, and Xin Xia. 2024. Reasoning runtime behavior of a program with llm: How far are we? arXiv preprint arXiv:2403.16437. — [arXiv:2403.16437](https://arxiv.org/abs/2403.16437) [find](https://scholar.google.com/scholar?q=Junkai+Chen%2C+Zhiyuan+Pan%2C+Xing+Hu%2C+Zhenhao+Li%2C+Ge+Li%2C+and+Xin+Xia%2E+2024%2E+Reasoning+runtime+behavior+of+a+program+with+llm%3A+How+far+are+we%3F+arXiv+preprint+arXiv%3A2403%2E16437%2E+%E2%80%94+%5BarXiv%3A2403%2E16437%5D%28https%3A%2F)
3. Nuo Chen, Zehua Li, Keqin Bao, Junyang Lin, and Dayiheng Liu. 2025. Chain of execution supervision promotes general reasoning in large language models. arXiv preprint arXiv:2510.23629. — [arXiv:2510.23629](https://arxiv.org/abs/2510.23629) [find](https://scholar.google.com/scholar?q=Nuo+Chen%2C+Zehua+Li%2C+Keqin+Bao%2C+Junyang+Lin%2C+and+Dayiheng+Liu%2E+2025%2E+Chain+of+execution+supervision+promotes+general+reasoning+in+large+language+models%2E+arXiv+preprint+arXiv%3A2510%2E23629%2E+%E2%80%94+%5BarXiv%3A2510%2E2)
4. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Teaching+large+language+models+to+self%2Ddebug%2E+arXiv+preprint+arXiv%3A2304%2E05128%2E)
5. Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, and 1 others. 2025. [Process reinforcement through implicit rewards](https://md.research.elamur.ai/free/en/papers/cui-2025-process-reinforcement-through-implicit-rewards/). arXiv preprint arXiv:2502.01456.
6. Yangruibo Ding, Jinjun Peng, Marcus Min, Gail Kaiser, Junfeng Yang, and Baishakhi Ray. 2024a. Semcoder: Training code language models with comprehensive semantics reasoning. Advances in Neural Information Processing Systems, 37:60275–60308. [find](https://scholar.google.com/scholar?q=Yangruibo+Ding%2C+Jinjun+Peng%2C+Marcus+Min%2C+Gail+Kaiser%2C+Junfeng+Yang%2C+and+Baishakhi+Ray%2E+2024a%2E+Semcoder%3A+Training+code+language+models+with+comprehensive+semantics+reasoning%2E+Advances+in+Neural+Informa)
7. Yangruibo Ding, Benjamin Steenhoek, Kexin Pei, Gail Kaiser, Wei Le, and Baishakhi Ray. 2024b. Traced: Execution-aware pre-training for source code. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pages 1–12. [find](https://scholar.google.com/scholar?q=Yangruibo+Ding%2C+Benjamin+Steenhoek%2C+Kexin+Pei%2C+Gail+Kaiser%2C+Wei+Le%2C+and+Baishakhi+Ray%2E+2024b%2E+Traced%3A+Execution%2Daware+pre%2Dtraining+for+source+code%2E+In+Proceedings+of+the+46th+IEEE%2FACM+International+Co)
8. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. [The llama 3 herd of models](https://md.research.elamur.ai/free/en/papers/grattafiori-2024-llama-3-herd-models/). arXiv e-prints, pages arXiv–2407.
9. Lishui Fan, Yu Zhang, Mouxiang Chen, and Zhongxin Liu. 2025. Posterior-grpo: Rewarding reasoning processes in code generation. arXiv preprint arXiv:2508.05170. — [arXiv:2508.05170](https://arxiv.org/abs/2508.05170) [find](https://scholar.google.com/scholar?q=Lishui+Fan%2C+Yu+Zhang%2C+Mouxiang+Chen%2C+and+Zhongxin+Liu%2E+2025%2E+Posterior%2Dgrpo%3A+Rewarding+reasoning+processes+in+code+generation%2E+arXiv+preprint+arXiv%3A2508%2E05170%2E+%E2%80%94+%5BarXiv%3A2508%2E05170%5D%28https%3A%2F%2Farxiv%2Eorg%2Fa)
10. Alex Gu, Wen-Ding Li, Naman Jain, Theo Olausson, Celine Lee, Koushik Sen, and Armando Solar-Lezama. 2024a. The counterfeit conundrum: Can code language models grasp the nuances of their incorrect generations? In Findings of the Association for Computational Linguistics: ACL 2024, pages 74–117. [find](https://scholar.google.com/scholar?q=Alex+Gu%2C+Wen%2DDing+Li%2C+Naman+Jain%2C+Theo+Olausson%2C+Celine+Lee%2C+Koushik+Sen%2C+and+Armando+Solar%2DLezama%2E+2024a%2E+The+counterfeit+conundrum%3A+Can+code+language+models+grasp+the+nuances+of+their+incorrect+gene)
11. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
12. Sonam Gupta, Yatin Nandwani, Asaf Yehudai, Dinesh Khandelwal, Dinesh Raghu, and Sachindra Joshi. 2025. Selective self-to-supervised fine-tuning for generalization in large language models. arXiv preprint arXiv:2502.08130. [find](https://scholar.google.com/scholar?q=Sonam+Gupta%2C+Yatin+Nandwani%2C+Asaf+Yehudai%2C+Dinesh+Khandelwal%2C+Dinesh+Raghu%2C+and+Sachindra+Joshi%2E+2025%2E+Selective+self%2Dto%2Dsupervised+fine%2Dtuning+for+generalization+in+large+language+models%2E+arXiv+prepr)
13. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and 1 others. 2021. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). arXiv preprint arXiv:2105.09938.
14. Siming Huang, Tianhao Cheng, Jason Klein Liu, Weidi Xu, Jiaran Hao, Liuyihan Song, Yang Xu, Jian Yang, Jiaheng Liu, Chenchen Zhang, and 1 others. 2025. [Opencoder: The open cookbook for top-tier code large language models](https://md.research.elamur.ai/free/en/papers/huang-2025-opencoder-open-cookbook-top-tier/). In Proceedings of the 63rd Annual Meeting of the Association for Computational…
15. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — [Qwen2.5-Coder Technical Report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
16. Rasha Ahmad Husein, Hala Aburajouh, and Cagatay Catal. 2025. Large language models for code completion: A systematic literature review. Computer Standards & Interfaces, 92:103917. [find](https://scholar.google.com/scholar?q=Rasha+Ahmad+Husein%2C+Hala+Aburajouh%2C+and+Cagatay+Catal%2E+2025%2E+Large+language+models+for+code+completion%3A+A+systematic+literature+review%2E+Computer+Standards+%26+Interfaces%2C+92%3A103917%2E)
17. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974.
18. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. [A survey on large language models for code generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/). arXiv preprint arXiv:2406.00515.
19. Dongwon Jung, Wenxuan Zhou, and Muhao Chen. 2025. Code execution as grounded supervision for llm reasoning. arXiv preprint arXiv:2506.10343. [find](https://scholar.google.com/scholar?q=Dongwon+Jung%2C+Wenxuan+Zhou%2C+and+Muhao+Chen%2E+2025%2E+Code+execution+as+grounded+supervision+for+llm+reasoning%2E+arXiv+preprint+arXiv%3A2506%2E10343%2E)
20. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024b. [Cruxeval: A benchmark for code reasoning, understanding and execution](https://md.research.elamur.ai/free/en/papers/gu-2024-cruxeval-benchmark-code-reasoning-understanding/). arXiv preprint arXiv:2401.03065.
21. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, and 1 others. 2022. The stack: 3 tb of permissively licensed source code. arXiv preprint arXiv:2211.15533. [find](https://scholar.google.com/scholar?q=Denis+Kocetkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Jia+Li%2C+Chenghao+Mou%2C+Carlos+Mu%C3%B1oz+Ferrandis%2C+Yacine+Jernite%2C+Margaret+Mitchell%2C+Sean+Hughes%2C+Thomas+Wolf%2C+and+1+others%2E+2022%2E+The+stack%3A+3+tb+of+permissi)
22. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. [Unixcoder: Unified cross-modal pre-training for code representation](https://md.research.elamur.ai/free/en/papers/guo-2022-unixcoder-unified-cross-modal-pre/). arXiv preprint arXiv:2203.03850.
23. Junlong Li, Daya Guo, Dejian Yang, Runxin Xu, Yu Wu, and Junxian He. 2025. [Codei/o: Condensing reasoning patterns via code input-output prediction](https://md.research.elamur.ai/free/en/papers/li-2025-codei-o-condensing-reasoning/). arXiv preprint arXiv:2502.07316.
24. Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852. [find](https://scholar.google.com/scholar?q=Rongao+Li%2C+Jie+Fu%2C+Bo%2DWen+Zhang%2C+Tao+Huang%2C+Zhihong+Sun%2C+Chen+Lyu%2C+Guang+Liu%2C+Zhi+Jin%2C+and+Ge+Li%2E+2023%2E+Taco%3A+Topics+in+algorithmic+code+generation+dataset%2E+arXiv+preprint+arXiv%3A2312%2E14852%2E)
25. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097.
26. Yihan Wang, Si Si, Daliang Li, Michal Lukasik, Felix Yu, Cho-Jui Hsieh, Inderjit S Dhillon, and Sanjiv Kumar. 2022. Two-stage llm fine-tuning with less specialization and more generalization. arXiv preprint arXiv:2211.00635. — [arXiv:2211.00635](https://arxiv.org/abs/2211.00635) [find](https://scholar.google.com/scholar?q=Yihan+Wang%2C+Si+Si%2C+Daliang+Li%2C+Michal+Lukasik%2C+Felix+Yu%2C+Cho%2DJui+Hsieh%2C+Inderjit+S+Dhillon%2C+and+Sanjiv+Kumar%2E+2022%2E+Two%2Dstage+llm+fine%2Dtuning+with+less+specialization+and+more+generalization%2E+arXiv+pr)
27. Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. 2023. Magicoder: Empowering code generation with oss-instruct. arXiv preprint arXiv:2312.02120. [find](https://scholar.google.com/scholar?q=Yuxiang+Wei%2C+Zhe+Wang%2C+Jiawei+Liu%2C+Yifeng+Ding%2C+and+Lingming+Zhang%2E+2023%2E+Magicoder%3A+Empowering+code+generation+with+oss%2Dinstruct%2E+arXiv+preprint+arXiv%3A2312%2E02120%2E)
28. Chenxiao Liu, Shuai Lu, Weizhu Chen, Daxin Jiang, Alexey Svyatkovskiy, Shengyu Fu, Neel Sundaresan, and Nan Duan. 2023a. Code execution with pre-trained language models. arXiv preprint arXiv:2305.05383. [find](https://scholar.google.com/scholar?q=Chenxiao+Liu%2C+Shuai+Lu%2C+Weizhu+Chen%2C+Daxin+Jiang%2C+Alexey+Svyatkovskiy%2C+Shengyu+Fu%2C+Neel+Sundaresan%2C+and+Nan+Duan%2E+2023a%2E+Code+execution+with+pre%2Dtrained+language+models%2E+arXiv+preprint+arXiv%3A2305%2E0538)
29. Danning Xie, Mingwei Zheng, Xuwei Liu, Jiannan Wang, Chengpeng Wang, Lin Tan, and Xiangyu Zhang. 2025. Core: Benchmarking llms code reasoning capabilities through static analysis tasks. arXiv preprint arXiv:2507.05269. — [arXiv:2507.05269](https://arxiv.org/abs/2507.05269) [find](https://scholar.google.com/scholar?q=Danning+Xie%2C+Mingwei+Zheng%2C+Xuwei+Liu%2C+Jiannan+Wang%2C+Chengpeng+Wang%2C+Lin+Tan%2C+and+Xiangyu+Zhang%2E+2025%2E+Core%3A+Benchmarking+llms+code+reasoning+capabilities+through+static+analysis+tasks%2E+arXiv+preprint)
30. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023b. [Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation](https://md.research.elamur.ai/free/en/papers/liu-2023-your-code-generated-chatgpt-really/). Advances in Neural Information Processing Systems, 36:21558– 21572.
31. Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, and Le Sun. 2025a. Cruxeval-x: A benchmark for multilingual code reasoning, understanding and execution. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume… [find](https://scholar.google.com/scholar?q=Ruiyang+Xu%2C+Jialun+Cao%2C+Yaojie+Lu%2C+Ming+Wen%2C+Hongyu+Lin%2C+Xianpei+Han%2C+Ben+He%2C+Shing%2DChi+Cheung%2C+and+Le+Sun%2E+2025a%2E+Cruxeval%2Dx%3A+A+benchmark+for+multilingual+code+reasoning%2C+understanding+and+execution%2E)
32. Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023c. Is your code generated by chat-GPT really correct? rigorous evaluation of large language models for code generation. In Thirty-seventh Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Jiawei+Liu%2C+Chunqiu+Steven+Xia%2C+Yuyao+Wang%2C+and+Lingming+Zhang%2E+2023c%2E+Is+your+code+generated+by+chat%2DGPT+really+correct%3F+rigorous+evaluation+of+large+language+models+for+code+generation%2E+In+Thirty%2Dse)
33. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2023%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E+a)
34. Ansong Ni, Miltiadis Allamanis, Arman Cohan, Yinlin Deng, Kensen Shi, Charles Sutton, and Pengcheng Yin. 2024. Next: Teaching large language models to reason about code execution. arXiv preprint arXiv:2404.14662. [find](https://scholar.google.com/scholar?q=Ansong+Ni%2C+Miltiadis+Allamanis%2C+Arman+Cohan%2C+Yinlin+Deng%2C+Kensen+Shi%2C+Charles+Sutton%2C+and+Pengcheng+Yin%2E+2024%2E+Next%3A+Teaching+large+language+models+to+reason+about+code+execution%2E+arXiv+preprint+arXiv)
35. Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. 2025. Codeforces. [https://huggingface.co/](https://huggingface.co/) datasets/open-r1/codeforces. [↗](https://huggingface.co/)
36. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2024. [Hybridflow: A flexible and efficient rlhf framework](https://md.research.elamur.ai/free/en/papers/sheng-2024-hybridflow-flexible-efficient-rlhf-framework/). arXiv preprint arXiv: 2409.19256.
37. Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. 2025. Reasoning gym: Reasoning environments for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2505.24760. — [arXiv:2505.24760](https://arxiv.org/abs/2505.24760) [find](https://scholar.google.com/scholar?q=Zafir+Stojanovski%2C+Oliver+Stanley%2C+Joe+Sharratt%2C+Richard+Jones%2C+Abdulhakeem+Adefioye%2C+Jean+Kaddour%2C+and+Andreas+K%C3%B6pf%2E+2025%2E+Reasoning+gym%3A+Reasoning+environments+for+reinforcement+learning+with+verifi)
38. Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, and Radha Poovendran. 2025b. Kodcode: A diverse, challenging, and verifiable synthetic dataset for coding. arXiv preprint arXiv:2503.02951. [find](https://scholar.google.com/scholar?q=Zhangchen+Xu%2C+Yang+Liu%2C+Yueqin+Yin%2C+Mingyuan+Zhou%2C+and+Radha+Poovendran%2E+2025b%2E+Kodcode%3A+A+diverse%2C+challenging%2C+and+verifiable+synthetic+dataset+for+coding%2E+arXiv+preprint+arXiv%3A2503%2E02951%2E)
39. He Ye, Matias Martinez, and Martin Monperrus. 2022. Neural program repair with execution-based backpropagation. In Proceedings of the 44th international conference on software engineering, pages 1506–1518. [find](https://scholar.google.com/scholar?q=He+Ye%2C+Matias+Martinez%2C+and+Martin+Monperrus%2E+2022%2E+Neural+program+repair+with+execution%2Dbased+backpropagation%2E+In+Proceedings+of+the+44th+international+conference+on+software+engineering%2C+pages+1506%E2%80%93)
40. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. [Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model?](https://md.research.elamur.ai/free/en/papers/yue-2025-reinforcement-learning-really-incentivize-reasoning/) arXiv preprint arXiv:2504.13837.
41. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024. [Llamafactory: Unified efficient fine-tuning of 100+ language models](https://md.research.elamur.ai/free/en/papers/zheng-2024-llamafactory-unified-efficient-fine-tuning/). arXiv preprint arXiv:2403.13372.
42. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, and 1 others. 2024. [Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2024-deepseek-coder-v2-breaking-barrier/). arXiv preprint arXiv:2406.11931.
43. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others. 2024. [Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/). arXiv preprint arXiv:2406.15877.
44. Qwen Team. 2024. Qwq: Reflect deeply on the boundaries of the unknown. Hugging Face. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2024%2E+Qwq%3A+Reflect+deeply+on+the+boundaries+of+the+unknown%2E+Hugging+Face%2E)

*Updated 31 October 2026*