---
title: "Seed Diffusion — paper by Song et al. (2025)"
description: "Discrete diffusion language models can generate many tokens in parallel, but two problems limit them… 32 references, cited by 28 papers here."
authors: ["Yuxuan Song", "Zheng Zhang", "Cheng Luo", "Pengyang Gao", "Fan Xia", "Hao Luo", "Zheng Li", "Yuehang Yang", "Hongli Yu", "Xingwei Qu", "Yuwei Fu", "Jing Su", "Ge Zhang", "Wenhao Huang", "Mingxuan Wang", "Lin Yan", "Xiaoying Jia", "Jingjing Liu", "Wei-Ying Ma", "Ya-Qin Zhang", "Yonghui Wu", "Hao Zhou"]
year: 2025
venue: "arXiv:2508.02193"
source: "https://arxiv.org/abs/2508.02193"
doi: "10.48550/arxiv.2508.02193"
arxiv: "2508.02193"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/song-2025-seed-diffusion-large-scale/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/song-2025-seed-diffusion-large-scale/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/) / Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

# Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference

**Authors:** Yuxuan Song, [Zheng Zhang](https://md.research.elamur.ai/free/en/authors/zheng-zhang/), [Cheng Luo](https://md.research.elamur.ai/free/en/authors/cheng-luo/), Pengyang Gao, Fan Xia, [Hao Luo](https://md.research.elamur.ai/free/en/authors/hao-luo/), [Zheng Li](https://md.research.elamur.ai/free/en/authors/zheng-li/), Yuehang Yang, Hongli Yu, [Xingwei Qu](https://md.research.elamur.ai/free/en/authors/xingwei-qu/), Yuwei Fu, [Jing Su](https://md.research.elamur.ai/free/en/authors/jing-su/), [Ge Zhang](https://md.research.elamur.ai/free/en/authors/ge-zhang/), [Wenhao Huang](https://md.research.elamur.ai/free/en/authors/wenhao-huang/), [Mingxuan Wang](https://md.research.elamur.ai/free/en/authors/mingxuan-wang/), Lin Yan, Xiaoying Jia, [Jingjing Liu](https://md.research.elamur.ai/free/en/authors/jingjing-liu/), Wei-Ying Ma, Ya-Qin Zhang, [Yonghui Wu](https://md.research.elamur.ai/free/en/authors/yonghui-wu/) and [Hao Zhou](https://md.research.elamur.ai/free/en/authors/hao-zhou/) · **Year:** 2025 · **Venue:** arXiv:2508.02193 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2508.02193) · [PDF](https://arxiv.org/pdf/2508.02193) · [DOI 10.48550/arxiv.2508.02193](https://doi.org/10.48550/arxiv.2508.02193) · [arXiv 2508.02193](https://arxiv.org/abs/2508.02193) · [OpenAlex](https://openalex.org/W4417102600)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [28](https://md.research.elamur.ai/free/en/papers/song-2025-seed-diffusion-large-scale/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary

Discrete diffusion language models can generate many tokens in parallel, but two problems limit them in practice. Learning from arbitrary token orders conflicts with the largely sequential structure of language, and iterative denoising adds latency that cancels the speed advantage. ByteDance Seed and Tsinghua AIR address both with Seed Diffusion Preview, a code-only dense Transformer trained on data from the Seed Coder pipeline.

Training happens in two stages. Most of it uses standard mask-based corruption. The final portion adds edit-based corruption (insertions, deletions and substitutions, scaled by Levenshtein distance), which pushes the model to re-check tokens that were left unmasked and reduces failures such as repetition. The model is then fine-tuned on generation trajectories selected by ELBO. Finally, on-policy learning cuts the number of decoding steps.

The model reaches 2,146 tokens/s on H20 GPUs. Its scores on code benchmarks are comparable to similarly sized autoregressive models, and it performs especially well on code editing. The authors argue this shows discrete diffusion is a credible route to fast LLM inference.

## Key points

- **Inference speed:** 2,146 tokens/s on H20 GPUs, claimed as a new speed–quality Pareto frontier ahead of Mercury Coder and Gemini Diffusion. The authors note that hardware and test conditions differ: Mercury was measured on H100s with a proprietary set, and Gemini on unknown hardware.
- **Two-stage curriculum (TSC):** the first 80% of diffusion steps use mask-only corruption. The last 20% add edit-based augmentation, with the edit ratio held within [0, 0.1]. The authors deliberately skip "Carry Over Unmasking" to avoid overconfidence in unmasked tokens.
- **Constrained-order training:** the pretrained model generates a pool of candidate trajectories. The best ones, chosen by ELBO, form a distilled dataset for fine-tuning, which narrows the any-order learning space.
- **On-policy diffusion learning:** this stage trains the model to shorten its sampled reverse trajectories while keeping output quality.
- **Evaluation:** the model was tested on HumanEval, MBPP, BigCodeBench, LiveCodeBench (1,055 problems, v1–v6), NaturalCodeBench (402 Python/Java problems), Aider (133 Exercism exercises) and CanItEdit (105 problems). The model size is not disclosed.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works

## Cite

Cite as: Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang, Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu and Hao Zhou. 2025. Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference. arXiv:2508.02193.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{song2025seed,
  title = {{Seed Diffusion: A Large-Scale Diffusion Language Model with High-Speed Inference}},
  author = {Yuxuan Song, Zheng Zhang, Cheng Luo, Pengyang Gao, Fan Xia, Hao Luo, Zheng Li, Yuehang Yang, Hongli Yu, Xingwei Qu, Yuwei Fu, Jing Su, Ge Zhang, Wenhao Huang, Mingxuan Wang, Lin Yan, Xiaoying Jia, Jingjing Liu, Wei-Ying Ma, Ya-Qin Zhang, Yonghui Wu and Hao Zhou},
  year = {2025},
  doi = {10.48550/arxiv.2508.02193},
  eprint = {2508.02193},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.02193}
}
```

- [← previous MultiSocial: Multilingual Benchmark of Machine-Generated Text Detection of Social-Media Texts](https://md.research.elamur.ai/free/en/papers/macko-2025-multisocial-multilingual-benchmark-machine-generated/)
- [next → Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above](https://md.research.elamur.ai/free/en/papers/balepur-2025-which-these-best-describes-multiple/)
- [↑ back to papers of 2025](https://md.research.elamur.ai/free/en/papers/2025/page/3/)

## References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samar Khanna, Siddhant Kharbanda, Shufan Li, Harshit Varma, Eric Wang, Sawyer Birnbaum, Ziyang Luo, Yanis Miraoui, Akash Palrecha, Stefano Ermon, et al. Mercury: Ultra-fast language models based on diffusion. arXiv e-prints, pages arXiv–2506, 2025. [find](https://scholar.google.com/scholar?q=Samar+Khanna%2C+Siddhant+Kharbanda%2C+Shufan+Li%2C+Harshit+Varma%2C+Eric+Wang%2C+Sawyer+Birnbaum%2C+Ziyang+Luo%2C+Yanis+Miraoui%2C+Akash+Palrecha%2C+Stefano+Ermon%2C+et+al%2E+Mercury%3A+Ultra%2Dfast+language+models+based+on+di)
2. Google DeepMind. [https://blog.google/technology/google-deepmind/gemini-diffusion/](https://blog.google/technology/google-deepmind/gemini-diffusion/). https://blog.google/ technology/google-deepmind/gemini-diffusion/, 2025. Accessed: 2024-07-24. [↗](https://blog.google/technology/google-deepmind/gemini-diffusion/)
3. Jonathan Ho, Ajay Jain, and Pieter Abbeel. arXiv:2006.11239, 2020. Denoising diffusion probabilistic models. arXiv preprint [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Ajay+Jain%2C+and+Pieter+Abbeel%2E+arXiv%3A2006%2E11239%2C+2020%2E+Denoising+diffusion+probabilistic+models%2E+arXiv+preprint)
4. Yang Song and Stefano Ermon. [Generative modeling by estimating gradients of the data distribution](https://md.research.elamur.ai/free/en/papers/song-2019-generative-modeling-estimating-gradients-data/). In Advances in Neural Information Processing Systems, pages 11918–11930, 2019.
5. Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. [Deep unsupervised learning using nonequilibrium thermodynamics](https://md.research.elamur.ai/free/en/papers/sohl-dickstein-2015-deep-unsupervised-learning-using-nonequilibrium/). arXiv preprint arXiv:1503.03585, 2015.
6. Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in neural information processing systems, 35:8633–8646, 2022. [find](https://scholar.google.com/scholar?q=Jonathan+Ho%2C+Tim+Salimans%2C+Alexey+Gritsenko%2C+William+Chan%2C+Mohammad+Norouzi%2C+and+David+J+Fleet%2E+Video+diffusion+models%2E+Advances+in+neural+information+processing+systems%2C+35%3A8633%E2%80%938646%2C+2022%2E)
7. Mihaly Varadi, Damian Bertoni, Paulyna Magana, Urmila Paramval, Ivanna Pidruchna, Malarvizhi Radhakrishnan, Maxim Tsenkov, Sreenath Nair, Milot Mirdita, Jingi Yeo, et al. Alphafold protein structure database in 2024: providing structure coverage for over 214 million protein sequences. Nucleic acids… [find](https://scholar.google.com/scholar?q=Mihaly+Varadi%2C+Damian+Bertoni%2C+Paulyna+Magana%2C+Urmila+Paramval%2C+Ivanna+Pidruchna%2C+Malarvizhi+Radhakrishnan%2C+Maxim+Tsenkov%2C+Sreenath+Nair%2C+Milot+Mirdita%2C+Jingi+Yeo%2C+et+al%2E+Alphafold+protein+structure+d)
8. Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez. Bayesian flow networks. arXiv preprint arXiv:2308.07037, 2023. [find](https://scholar.google.com/scholar?q=Alex+Graves%2C+Rupesh+Kumar+Srivastava%2C+Timothy+Atkinson%2C+and+Faustino+Gomez%2E+Bayesian+flow+networks%2E+arXiv+preprint+arXiv%3A2308%2E07037%2C+2023%2E)
9. Sander Dieleman, Laurent Sartran, Arman Roshannai, Nikolay Savinov, Yaroslav Ganin, Pierre H Richemond, Arnaud Doucet, Robin Strudel, Chris Dyer, Conor Durkan, et al. Continuous diffusion for categorical data. arXiv preprint arXiv:2211.15089, 2022. — [arXiv:2211.15089](https://arxiv.org/abs/2211.15089) [find](https://scholar.google.com/scholar?q=Sander+Dieleman%2C+Laurent+Sartran%2C+Arman+Roshannai%2C+Nikolay+Savinov%2C+Yaroslav+Ganin%2C+Pierre+H+Richemond%2C+Arnaud+Doucet%2C+Robin+Strudel%2C+Chris+Dyer%2C+Conor+Durkan%2C+et+al%2E+Continuous+diffusion+for+categori)
10. Ishaan Gulrajani and Tatsunori B Hashimoto. Likelihood-based diffusion language models. Advances in Neural Information Processing Systems, 36, 2024. [find](https://scholar.google.com/scholar?q=Ishaan+Gulrajani+and+Tatsunori+B+Hashimoto%2E+Likelihood%2Dbased+diffusion+language+models%2E+Advances+in+Neural+Information+Processing+Systems%2C+36%2C+2024%2E)
11. Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. [Structured denoising diffusion models in discrete state-spaces](https://md.research.elamur.ai/free/en/papers/austin-2021-structured-denoising-diffusion-models-discrete/). Advances in Neural Information Processing Systems, 34:17981–17993, 2021.
12. Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T Chiu, Alexander Rush, and Volodymyr Kuleshov. [Simple and effective masked diffusion language models](https://md.research.elamur.ai/free/en/papers/sahoo-2024-simple-effective-masked-diffusion-language/). arXiv preprint arXiv:2406.07524, 2024.
13. Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis K Titsias. Simplified and generalized masked diffusion for discrete data. arXiv preprint arXiv:2406.04329, 2024. [find](https://scholar.google.com/scholar?q=Jiaxin+Shi%2C+Kehang+Han%2C+Zhe+Wang%2C+Arnaud+Doucet%2C+and+Michalis+K+Titsias%2E+Simplified+and+generalized+masked+diffusion+for+discrete+data%2E+arXiv+preprint+arXiv%3A2406%2E04329%2C+2024%2E)
14. Jingyang Ou, Shen Nie, Kaiwen Xue, Fengqi Zhu, Jiacheng Sun, Zhenguo Li, and Chongxuan Li. Your absorbing discrete diffusion secretly models the conditional distributions of clean data. arXiv preprint arXiv:2406.03736, 2024. — [arXiv:2406.03736](https://arxiv.org/abs/2406.03736) [find](https://scholar.google.com/scholar?q=Jingyang+Ou%2C+Shen+Nie%2C+Kaiwen+Xue%2C+Fengqi+Zhu%2C+Jiacheng+Sun%2C+Zhenguo+Li%2C+and+Chongxuan+Li%2E+Your+absorbing+discrete+diffusion+secretly+models+the+conditional+distributions+of+clean+data%2E+arXiv+preprint)
15. Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li. [Large language diffusion models](https://md.research.elamur.ai/free/en/papers/nie-2025-large-language-diffusion-models/). arXiv preprint arXiv:2502.09992, 2025.
16. Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. [Glancing transformer for non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/qian-2021-glancing-transformer-non-autoregressive-neural/). In the 59th Annual Meeting of the Association for Computational Linguistics (ACL), July 2021.
17. Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. Directed acyclic transformer for non-autoregressive machine translation. In International Conference on Machine Learning, pages 9410–9428. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Fei+Huang%2C+Hao+Zhou%2C+Yang+Liu%2C+Hang+Li%2C+and+Minlie+Huang%2E+Directed+acyclic+transformer+for+non%2Dautoregressive+machine+translation%2E+In+International+Conference+on+Machine+Learning%2C+pages+9410%E2%80%939428%2E+PML)
18. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. [Mask-predict: Parallel decoding of conditional masked language models](https://md.research.elamur.ai/free/en/papers/ghazvininejad-2019-mask-predict-parallel-decoding-conditional/). arXiv preprint arXiv:1904.09324, 2019.
19. Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion language modeling by estimating the ratios of the data distribution. 2023. [find](https://scholar.google.com/scholar?q=Aaron+Lou%2C+Chenlin+Meng%2C+and+Stefano+Ermon%2E+Discrete+diffusion+language+modeling+by+estimating+the+ratios+of+the+data+distribution%2E+2023%2E)
20. Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, et al. [Seed-coder: Let the code model curate data for itself](https://md.research.elamur.ai/free/en/papers/bytedance-seed-2025-seed-coder-let-code/). arXiv preprint arXiv:2506.03524, 2025.
21. Lihua Qian, Mingxuan Wang, Yang Liu, and Hao Zhou. Diffusion glancing transformer for parallel sequence-to-sequence learning. In Kevin Duh, Helena Gomez, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Hu… [find](https://scholar.google.com/scholar?q=Lihua+Qian%2C+Mingxuan+Wang%2C+Yang+Liu%2C+and+Hao+Zhou%2E+Diffusion+glancing+transformer+for+parallel+sequence%2Dto%2Dsequence+learning%2E+In+Kevin+Duh%2C+Helena+Gomez%2C+and+Steven+Bethard%2C+editors%2C+Proceedings+of+th)
22. Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. arXiv preprint arXiv:2110.02037, 2021. — [arXiv:2110.02037](https://arxiv.org/abs/2110.02037) [find](https://scholar.google.com/scholar?q=Emiel+Hoogeboom%2C+Alexey+A+Gritsenko%2C+Jasmijn+Bastings%2C+Ben+Poole%2C+Rianne+van+den+Berg%2C+and+Tim+Salimans%2E+Autoregressive+diffusion+models%2E+arXiv+preprint+arXiv%3A2110%2E02037%2C+2021%2E+%E2%80%94+%5BarXiv%3A2110%2E02037%5D%28ht)
23. Ning Miao, Yuxuan Song, Hao Zhou, and Lei Li. Do you have the right scissors? tailoring pre-trained language models via Monte-Carlo methods. In the 58th Annual Meeting of the Association for Computational Linguistics (ACL) - short papers, July 2020. [find](https://scholar.google.com/scholar?q=Ning+Miao%2C+Yuxuan+Song%2C+Hao+Zhou%2C+and+Lei+Li%2E+Do+you+have+the+right+scissors%3F+tailoring+pre%2Dtrained+language+models+via+Monte%2DCarlo+methods%2E+In+the+58th+Annual+Meeting+of+the+Association+for+Computati)
24. Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1–62, 2020. [find](https://scholar.google.com/scholar?q=Shakir+Mohamed%2C+Mihaela+Rosca%2C+Michael+Figurnov%2C+and+Andriy+Mnih%2E+Monte+carlo+gradient+estimation+in+machine+learning%2E+Journal+of+Machine+Learning+Research%2C+21%28132%29%3A1%E2%80%9362%2C+2020%2E)
25. Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. [Non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). In ICLR, 2018.
26. Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, and Volodymyr Kuleshov. Block diffusion: Interpolating between autoregressive and diffusion language models. arXiv preprint arXiv:2503.09573, 2025. — [arXiv:2503.09573](https://arxiv.org/abs/2503.09573) [find](https://scholar.google.com/scholar?q=Marianne+Arriola%2C+Aaron+Gokaslan%2C+Justin+T+Chiu%2C+Zhihan+Yang%2C+Zhixuan+Qi%2C+Jiaqi+Han%2C+Subham+Sekhar+Sahoo%2C+and+Volodymyr+Kuleshov%2E+Block+diffusion%3A+Interpolating+between+autoregressive+and+diffusion+la)
27. Jinyi Hu, Shengding Hu, Yuxuan Song, Yufei Huang, Mingxuan Wang, Hao Zhou, Zhiyuan Liu, Wei-Ying Ma, and Maosong Sun. Acdit: Interpolating autoregressive conditional modeling and diffusion transformer. arXiv preprint arXiv:2412.07720, 2024. — [arXiv:2412.07720](https://arxiv.org/abs/2412.07720) [find](https://scholar.google.com/scholar?q=Jinyi+Hu%2C+Shengding+Hu%2C+Yuxuan+Song%2C+Yufei+Huang%2C+Mingxuan+Wang%2C+Hao+Zhou%2C+Zhiyuan+Liu%2C+Wei%2DYing+Ma%2C+and+Maosong+Sun%2E+Acdit%3A+Interpolating+autoregressive+conditional+modeling+and+diffusion+transformer)
28. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. [Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/). arXiv preprint arXiv:2406.15877, 2024.
29. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974, 2024.
30. Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, et al. Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868, 2022. [find](https://scholar.google.com/scholar?q=Ben+Athiwaratkun%2C+Sanjay+Krishna+Gouda%2C+Zijian+Wang%2C+Xiaopeng+Li%2C+Yuchen+Tian%2C+Ming+Tan%2C+Wasi+Uddin+Ahmad%2C+Shiqi+Wang%2C+Qing+Sun%2C+Mingyue+Shang%2C+et+al%2E+Multi%2Dlingual+evaluation+of+code+generation+model)
31. Shudan Zhang, Hanlin Zhao, Xiao Liu, Qinkai Zheng, Zehan Qi, Xiaotao Gu, Xiaohan Zhang, Yuxiao Dong, and Jie Tang. Naturalcodebench: Examining coding performance mismatch on humaneval and natural user prompts. arXiv preprint arXiv:2405.04520, 2024. — [arXiv:2405.04520](https://arxiv.org/abs/2405.04520) [find](https://scholar.google.com/scholar?q=Shudan+Zhang%2C+Hanlin+Zhao%2C+Xiao+Liu%2C+Qinkai+Zheng%2C+Zehan+Qi%2C+Xiaotao+Gu%2C+Xiaohan+Zhang%2C+Yuxiao+Dong%2C+and+Jie+Tang%2E+Naturalcodebench%3A+Examining+coding+performance+mismatch+on+humaneval+and+natural+user)
32. Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Jacob Ginesin, Edward Berman, George Chakhnashvili, Anton Lozhkov, Carolyn Jane Anderson, et al. Can it edit? evaluating the ability of large language models to follow code editing instructions. arXiv preprint arXiv:2312.12450,… — [arXiv:2312.12450](https://arxiv.org/abs/2312.12450) [find](https://scholar.google.com/scholar?q=Federico+Cassano%2C+Luisa+Li%2C+Akul+Sethi%2C+Noah+Shinn%2C+Abby+Brennan%2DJones%2C+Jacob+Ginesin%2C+Edward+Berman%2C+George+Chakhnashvili%2C+Anton+Lozhkov%2C+Carolyn+Jane+Anderson%2C+et+al%2E+Can+it+edit%3F+evaluating+the+abi)

*Updated 31 October 2026*