---
title: "code-transformed — paper by Xu et al. (2026)"
description: "The authors present this as possibly the first large-scale empirical evidence that LLMs are reshaping human programming style. 58 references."
authors: ["Yuliang Xu", "Siming Huang", "Mingmeng Geng", "Yao Wan", "Xuanhua Shi", "Dongping Chen"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: EACL 2026"
source: "https://aclanthology.org/2026.findings-eacl.290/"
doi: "10.18653/v1/2026.findings-eacl.290"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/xu-2026-code-transformed-influence-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/xu-2026-code-transformed-influence-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/) / code-transformed: The Influence of Large Language Models on Code

# code-transformed: The Influence of Large Language Models on Code

**Authors:** Yuliang Xu, [Siming Huang](https://md.research.elamur.ai/free/en/authors/siming-huang/), Mingmeng Geng, [Yao Wan](https://md.research.elamur.ai/free/en/authors/yao-wan/), Xuanhua Shi and [Dongping Chen](https://md.research.elamur.ai/free/en/authors/dongping-chen/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: EACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-eacl.290/) · [PDF](https://aclanthology.org/2026.findings-eacl.290.pdf) · [DOI 10.18653/v1/2026.findings-eacl.290](https://doi.org/10.18653/v1/2026.findings-eacl.290) · [OpenAlex](https://openalex.org/W7140081073)

**Citations:** 0 citations (OpenAlex, 2026-10-14)

## Summary
The paper asks whether the spread of LLM coding assistants has left measurable traces in real-world code style. The authors take two approaches. First, they simulate LLM use. Human Codeforces solutions from Code4Bench, written before 2020, are compared with code that six LLMs either produced from scratch or rewrote from human solutions. The comparison covers naming formats, cyclomatic complexity, Halstead metrics, maintainability indices and cosine/Jaccard similarity. Second, they track the same signals over time in about 22k GitHub repositories tied to arXiv papers from 2020–2025. Naming conventions favoured by LLMs, especially longer snake_case identifiers, become clearly more common in those repositories. Complexity metrics show no clear temporal trend. Rewritten code stays fairly close to the human original, while directly generated code diverges more. A further analysis of model reasoning traces shows that LLMs often consider algorithms unrelated to the intended solution. The authors present this as possibly the first large-scale empirical evidence that LLMs are reshaping human programming style. They note that this matters for monitoring, attribution and the wider societal effects of AI-assisted coding.

## Key points
- **GitHub dataset:** 22,074 repositories and 1,032,016 source files linked to arXiv papers from Q1 2020 to Q3 2025. Each repository is labelled by language (Python or C/C++) and domain (cs vs. non-cs).
- **Naming trend:** in Python, the share of snake_case function names grew from 40.7% in Q1 2023 to 49.8% in Q3 2025. Names such as `max_length` are markedly more frequent in LLM output than in human code.
- **Simulation setup:**
  - Problems: 200 Code4Bench problems, 50 from each of four difficulty bands (800–1199 up to 2000+), covering ten algorithm tags.
  - Models: GPT-4.1, Gemini-2.0-flash, DeepSeek-R1-Distill-Qwen-32B, Llama-4-Maverick, Qwen3-32B and Gemma-3-27B.
  - Strategies: direct generation and rewriting of human solutions.
- **Complexity:** LLM rewriting lowers cyclomatic complexity for Python code, but no such trend appears in the GitHub data.
- **Reasoning analysis:** the error rate in models' reasoning traces generally exceeds the match rate with the intended algorithm tags. Both rates are higher for C/C++ and for harder problems. Data and code are released at github.com/ignorancex/LLM_code.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huazhong University of Science and Technology](https://md.research.elamur.ai/free/en/universities/huazhong-university-of-science-and-technology/) — University, Wuhan, China · 59 works · 8,310 citations

📚 Shelves (3): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 131 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Yuliang Xu, Siming Huang, Mingmeng Geng, Yao Wan, Xuanhua Shi and Dongping Chen. 2026. code-transformed: The Influence of Large Language Models on Code. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{xu2026code,
  title = {{code-transformed: The Influence of Large Language Models on Code}},
  author = {Yuliang Xu, Siming Huang, Mingmeng Geng, Yao Wan, Xuanhua Shi and Dongping Chen},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.290},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.290}
}
```

- [← previous Code over Words: Overcoming Semantic Inertia via Code-Grounded Reasoning](https://md.research.elamur.ai/free/en/papers/xu-2026-code-over-words-overcoming-semantic/)
- [next → CodecMOS-Accent: A MOS Benchmark of Resynthesized and TTS Speech from Neural Codecs Across English Accents](https://md.research.elamur.ai/free/en/papers/huang-2026-codecmos-accent-mos-benchmark-resynthesized/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/27/)

## References

58 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anthropic. 2024. Anthropic: Introducing claude 3.5 sonnet. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Anthropic%3A+Introducing+claude+3%2E5+sonnet%2E)
2. Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, and 1 others. 2025. [Llama-nemotron: Efficient reasoning models](https://md.research.elamur.ai/free/en/papers/bercovich-2025-llama-nemotron-efficient-reasoning-models/). arXiv preprint arXiv:2505.00949.
3. Luana Bulla, Alessandro Midolo, Misael Mongiovì, and Emiliano Tramontana. 2024. Ex-code: A robust and explainable model to detect ai-generated code. Information, 15(12):819. [find](https://scholar.google.com/scholar?q=Luana+Bulla%2C+Alessandro+Midolo%2C+Misael+Mongiov%C3%AC%2C+and+Emiliano+Tramontana%2E+2024%2E+Ex%2Dcode%3A+A+robust+and+explainable+model+to+detect+ai%2Dgenerated+code%2E+Information%2C+15%2812%29%3A819%2E)
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
5. Wentao Chen, Lizhe Zhang, Li Zhong, Letian Peng, Zilong Wang, and Jingbo Shang. 2025. Memorize or generalize? evaluating llm code generation with evolved questions. arXiv preprint arXiv:2503.02296. — [arXiv:2503.02296](https://arxiv.org/abs/2503.02296) [find](https://scholar.google.com/scholar?q=Wentao+Chen%2C+Lizhe+Zhang%2C+Li+Zhong%2C+Letian+Peng%2C+Zilong+Wang%2C+and+Jingbo+Shang%2E+2025%2E+Memorize+or+generalize%3F+evaluating+llm+code+generation+with+evolved+questions%2E+arXiv+preprint+arXiv%3A2503%2E02296%2E+%E2%80%94+)
6. Kyle Daigle and GitHub Staff. 2024. Survey: The ai wave continues to grow on software development teams. GitHub Blog. Available online: https://github. blog/news-insights/research/survey-ai-wave-grows/\# key-survey-findings. [find](https://scholar.google.com/scholar?q=Kyle+Daigle+and+GitHub+Staff%2E+2024%2E+Survey%3A+The+ai+wave+continues+to+grow+on+software+development+teams%2E+GitHub+Blog%2E+Available+online%3A+https%3A%2F%2Fgithub%2E+blog%2Fnews%2Dinsights%2Fresearch%2Fsurvey%2Dai%2Dwave%2Dgrows)
7. Google DeepMind. 2024. Introducing gemini 2.0: Our new ai model for the agentic era. [find](https://scholar.google.com/scholar?q=Google+DeepMind%2E+2024%2E+Introducing+gemini+2%2E0%3A+Our+new+ai+model+for+the+agentic+era%2E)
8. Basak Demirok and Mucahid Kutlu. 2024. Aigcode-set: A new annotated dataset for ai generated code detection. arXiv preprint arXiv:2412.16594. — [arXiv:2412.16594](https://arxiv.org/abs/2412.16594) [find](https://scholar.google.com/scholar?q=Basak+Demirok+and+Mucahid+Kutlu%2E+2024%2E+Aigcode%2Dset%3A+A+new+annotated+dataset+for+ai+generated+code+detection%2E+arXiv+preprint+arXiv%3A2412%2E16594%2E+%E2%80%94+%5BarXiv%3A2412%2E16594%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2412%2E16594%29)
9. Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. 2025. Codescore: Evaluating code generation by learning code execution. ACM Transactions on Software Engineering and Methodology, 34(3):1–22. [find](https://scholar.google.com/scholar?q=Yihong+Dong%2C+Jiazheng+Ding%2C+Xue+Jiang%2C+Ge+Li%2C+Zhuo+Li%2C+and+Zhi+Jin%2E+2025%2E+Codescore%3A+Evaluating+code+generation+by+learning+code+execution%2E+ACM+Transactions+on+Software+Engineering+and+Methodology%2C+34)
10. Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, and 1 others. 2024. What’s wrong with your code generated by large language models? an extensive study. arXiv preprint arXiv:2407.06153. — [arXiv:2407.06153](https://arxiv.org/abs/2407.06153) [find](https://scholar.google.com/scholar?q=Shihan+Dou%2C+Haoxiang+Jia%2C+Shenxi+Wu%2C+Huiyuan+Zheng%2C+Weikang+Zhou%2C+Muling+Wu%2C+Mingxu+Chai%2C+Jessica+Fan%2C+Caishuang+Huang%2C+Yunbo+Tao%2C+and+1+others%2E+2024%2E+What%E2%80%99s+wrong+with+your+code+generated+by+large+la)
11. Steffen Eger, Yong Cao, Jennifer D’Souza, Andreas Geiger, Christian Greisinger, Stephanie Gross, Yufang Hou, Brigitte Krenn, Anne Lauscher, Yizhi Li, and 1 others. 2025. [Transforming science with large language models: A survey on ai-assisted scientific discovery, experimentation, content generation…](https://md.research.elamur.ai/free/en/papers/eger-2025-transforming-science-large-language-models/)
12. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and 1 others. 2020. [Codebert: A pre-trained model for programming and natural languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/). arXiv preprint arXiv:2002.08155.
13. Mingmeng Geng, Caixi Chen, Yanru Wu, Dongping Chen, Yao Wan, and Pan Zhou. 2024. The impact of large language models in academia: from writing to speaking. arXiv preprint arXiv:2409.13686. [find](https://scholar.google.com/scholar?q=Mingmeng+Geng%2C+Caixi+Chen%2C+Yanru+Wu%2C+Dongping+Chen%2C+Yao+Wan%2C+and+Pan+Zhou%2E+2024%2E+The+impact+of+large+language+models+in+academia%3A+from+writing+to+speaking%2E+arXiv+preprint+arXiv%3A2409%2E13686%2E)
14. Mingmeng Geng and Roberto Trotta. 2024. Is chatgpt transforming academics’ writing style? arXiv preprint arXiv:2404.08627. — [arXiv:2404.08627](https://arxiv.org/abs/2404.08627) [find](https://scholar.google.com/scholar?q=Mingmeng+Geng+and+Roberto+Trotta%2E+2024%2E+Is+chatgpt+transforming+academics%E2%80%99+writing+style%3F+arXiv+preprint+arXiv%3A2404%2E08627%2E+%E2%80%94+%5BarXiv%3A2404%2E08627%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2404%2E08627%29)
15. Jay Graylin, Randy K SMITH, HALE David, Nicholas A KRAFT, WARD Charles, and 1 others. 2009. Cyclomatic complexity and lines of code: Empirical evidence of a stable linear relationship. Journal of Software Engineering and Applications, 2(3):137–143. [find](https://scholar.google.com/scholar?q=Jay+Graylin%2C+Randy+K+SMITH%2C+HALE+David%2C+Nicholas+A+KRAFT%2C+WARD+Charles%2C+and+1+others%2E+2009%2E+Cyclomatic+complexity+and+lines+of+code%3A+Empirical+evidence+of+a+stable+linear+relationship%2E+Journal+of+Soft)
16. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). arXiv preprint arXiv:2501.12948.
17. Andrea Gurioli, Maurizio Gabbrielli, and Stefano Zacchiroli. 2024. Is this you, llm? recognizing ai-written programs with multilingual code stylometry. arXiv preprint arXiv:2412.14611. — [arXiv:2412.14611](https://arxiv.org/abs/2412.14611) [find](https://scholar.google.com/scholar?q=Andrea+Gurioli%2C+Maurizio+Gabbrielli%2C+and+Stefano+Zacchiroli%2E+2024%2E+Is+this+you%2C+llm%3F+recognizing+ai%2Dwritten+programs+with+multilingual+code+stylometry%2E+arXiv+preprint+arXiv%3A2412%2E14611%2E+%E2%80%94+%5BarXiv%3A2412%2E1)
18. T Hariprasad, G Vidhyagaran, K Seenu, and Chan-drasegar Thirumalai. 2017. Software complexity analysis using halstead metrics. In 2017 international conference on trends in electronics and informatics (ICEI), pages 1109–1113. IEEE. [find](https://scholar.google.com/scholar?q=T+Hariprasad%2C+G+Vidhyagaran%2C+K+Seenu%2C+and+Chan%2Ddrasegar+Thirumalai%2E+2017%2E+Software+complexity+analysis+using+halstead+metrics%2E+In+2017+international+conference+on+trends+in+electronics+and+informatics)
19. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — [Qwen2.5-Coder Technical Report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/)
20. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. [Gpt-4o system card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276.
21. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) arXiv preprint arXiv:2310.06770.
22. Kai Korpimies, Antti Laaksonen, and Matti Luukkainen. 2024. Unrestricted use of llms in a software project course: Student perceptions on learning and impact on course performance. In Proceedings of the 24th Koli Calling International Conference on Computing Education Research, pages 1–7. [find](https://scholar.google.com/scholar?q=Kai+Korpimies%2C+Antti+Laaksonen%2C+and+Matti+Luukkainen%2E+2024%2E+Unrestricted+use+of+llms+in+a+software+project+course%3A+Student+perceptions+on+learning+and+impact+on+course+performance%2E+In+Proceedings+of+t)
23. Ruimiao Li, Manli Li, and Weifeng Qiao. 2025. Engineering students’ use of large language model tools: An empirical study based on a survey of students from 12 universities. Education Sciences, 15(3):280. [find](https://scholar.google.com/scholar?q=Ruimiao+Li%2C+Manli+Li%2C+and+Weifeng+Qiao%2E+2025%2E+Engineering+students%E2%80%99+use+of+large+language+model+tools%3A+An+empirical+study+based+on+a+survey+of+students+from+12+universities%2E+Education+Sciences%2C+15%283%29%3A)
24. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097.
25. Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024a. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM international conference on software engineering, pages 1–13. [find](https://scholar.google.com/scholar?q=Jenny+T+Liang%2C+Chenyang+Yang%2C+and+Brad+A+Myers%2E+2024a%2E+A+large%2Dscale+survey+on+the+usability+of+ai+programming+assistants%3A+Successes+and+challenges%2E+In+Proceedings+of+the+46th+IEEE%2FACM+international+c)
26. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, and 1 others. 2024b. [Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews](https://md.research.elamur.ai/free/en/papers/liang-2024-monitoring-ai-modified-content/). arXiv preprint arXiv:2403.07183.
27. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. [find](https://scholar.google.com/scholar?q=Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+and+1+others%2E+2024%2E+Deepseek%2Dv3+technical+report%2E+arXiv+preprint+arXiv%3A2412)
28. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, and 1 others. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664. [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+and+1+others%2E+2021%2E+Codexglue%3A+A+machine+learning+benchmark+dataset)
29. Amirabbas Majd, Mojtaba Vahidi-Asl, Alireza Khalilian, Ahmad Baraani-Dastjerdi, and Bahman Zamani. 2019. Code4bench: A multidimensional benchmark of codeforces data for different program analysis techniques. Journal of Computer Languages, 53:38–52. [find](https://scholar.google.com/scholar?q=Amirabbas+Majd%2C+Mojtaba+Vahidi%2DAsl%2C+Alireza+Khalilian%2C+Ahmad+Baraani%2DDastjerdi%2C+and+Bahman+Zamani%2E+2019%2E+Code4bench%3A+A+multidimensional+benchmark+of+codeforces+data+for+different+program+analysis+tech)
30. Dung Nguyen Manh, Thang Phan Chau, Nam Le Hai, Thong T Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui. 2024. Codemmlu: A multi-task benchmark for assessing code understanding capabilities of codellms. arXiv preprint arXiv:2410.01999. — [arXiv:2410.01999](https://arxiv.org/abs/2410.01999) [find](https://scholar.google.com/scholar?q=Dung+Nguyen+Manh%2C+Thang+Phan+Chau%2C+Nam+Le+Hai%2C+Thong+T+Doan%2C+Nam+V+Nguyen%2C+Quang+Pham%2C+and+Nghi+DQ+Bui%2E+2024%2E+Codemmlu%3A+A+multi%2Dtask+benchmark+for+assessing+code+understanding+capabilities+of+codellms)
31. Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering, 2(4):308–320. [find](https://scholar.google.com/scholar?q=Thomas+J+McCabe%2E+1976%2E+A+complexity+measure%2E+IEEE+Transactions+on+software+Engineering%2C+2%284%29%3A308%E2%80%93320%2E)
32. AI Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. [https://ai.meta](https://ai.meta). com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025. [↗](https://ai.meta)
33. OpenAI. 2025. Introducing gpt-4.1 in the api. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025%2E+Introducing+gpt%2D4%2E1+in+the+api%2E)
34. Shinwoo Park, Hyundong Jin, Jeong-won Cha, and Yo-Sub Han. 2025. Detection of llm-paraphrased code and identification of the responsible llm using coding style features. arXiv preprint arXiv:2502.17749. — [arXiv:2502.17749](https://arxiv.org/abs/2502.17749) [find](https://scholar.google.com/scholar?q=Shinwoo+Park%2C+Hyundong+Jin%2C+Jeong%2Dwon+Cha%2C+and+Yo%2DSub+Han%2E+2025%2E+Detection+of+llm%2Dparaphrased+code+and+identification+of+the+responsible+llm+using+coding+style+features%2E+arXiv+preprint+arXiv%3A2502%2E1774)
35. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Communications of the ACM, 68(2):96–105. [find](https://scholar.google.com/scholar?q=Hammond+Pearce%2C+Baleegh+Ahmad%2C+Benjamin+Tan%2C+Brendan+Dolan%2DGavitt%2C+and+Ramesh+Karri%2E+2025%2E+Asleep+at+the+keyboard%3F+assessing+the+security+of+github+copilot%E2%80%99s+code+contributions%2E+Communications+of+the+)
36. Alan Peslak and Lisa Kovalchick. 2024. Ai for coders: An analysis of the usage of chatgpt and github copilot. Issues in Information Systems, 25(4):252–260. [find](https://scholar.google.com/scholar?q=Alan+Peslak+and+Lisa+Kovalchick%2E+2024%2E+Ai+for+coders%3A+An+analysis+of+the+usage+of+chatgpt+and+github+copilot%2E+Issues+in+Information+Systems%2C+25%284%29%3A252%E2%80%93260%2E)
37. Musfiqur Rahman, SayedHassan Khatoonabadi, Ahmad Abdellatif, and Emad Shihab. 2024. Automatic detection of llm-generated code: A case study of claude 3 haiku. arXiv preprint arXiv:2409.01382. — [arXiv:2409.01382](https://arxiv.org/abs/2409.01382) [find](https://scholar.google.com/scholar?q=Musfiqur+Rahman%2C+SayedHassan+Khatoonabadi%2C+Ahmad+Abdellatif%2C+and+Emad+Shihab%2E+2024%2E+Automatic+detection+of+llm%2Dgenerated+code%3A+A+case+study+of+claude+3+haiku%2E+arXiv+preprint+arXiv%3A2409%2E01382%2E+%E2%80%94+%5BarXiv)
38. Sanka Rasnayaka, Guanlin Wang, Ridwan Shariffdeen, and Ganesh Neelakanta Iyer. 2024. An empirical study on usage and perceptions of llms in a software engineering project. In Proceedings of the 1st International Workshop on Large Language Models for Code, pages 111–118. [find](https://scholar.google.com/scholar?q=Sanka+Rasnayaka%2C+Guanlin+Wang%2C+Ridwan+Shariffdeen%2C+and+Ganesh+Neelakanta+Iyer%2E+2024%2E+An+empirical+study+on+usage+and+perceptions+of+llms+in+a+software+engineering+project%2E+In+Proceedings+of+the+1st+In)
39. Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. 2024. Between lines of code: Unraveling the distinct patterns of machine and human programmers. arXiv preprint arXiv:2401.06461. — [arXiv:2401.06461](https://arxiv.org/abs/2401.06461) [find](https://scholar.google.com/scholar?q=Yuling+Shi%2C+Hongyu+Zhang%2C+Chengcheng+Wan%2C+and+Xiaodong+Gu%2E+2024%2E+Between+lines+of+code%3A+Unraveling+the+distinct+patterns+of+machine+and+human+programmers%2E+arXiv+preprint+arXiv%3A2401%2E06461%2E+%E2%80%94+%5BarXiv%3A240)
40. Hyunjae Suh, Mahan Tafreshipour, Jiawei Li, Adithya Bhattiprolu, and Iftekhar Ahmed. 2024. An empirical study on automatically detecting ai-generated source code: How far are we? arXiv preprint arXiv:2411.04299. — [arXiv:2411.04299](https://arxiv.org/abs/2411.04299) [find](https://scholar.google.com/scholar?q=Hyunjae+Suh%2C+Mahan+Tafreshipour%2C+Jiawei+Li%2C+Adithya+Bhattiprolu%2C+and+Iftekhar+Ahmed%2E+2024%2E+An+empirical+study+on+automatically+detecting+ai%2Dgenerated+source+code%3A+How+far+are+we%3F+arXiv+preprint+arXiv%3A)
41. Zhenyu Xu and Victor S Sheng. 2024. Detecting ai-generated code assignments using perplexity of large language models. In Proceedings of the aaai conference on artificial intelligence, volume 38, pages 23155–23162. [find](https://scholar.google.com/scholar?q=Zhenyu+Xu+and+Victor+S+Sheng%2E+2024%2E+Detecting+ai%2Dgenerated+code+assignments+using+perplexity+of+large+language+models%2E+In+Proceedings+of+the+aaai+conference+on+artificial+intelligence%2C+volume+38%2C+page)
42. Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. 2025. Bugs in large language models generated code: An empirical study. Empirical Software Engineering, 30(3):1–48. [find](https://scholar.google.com/scholar?q=Florian+Tambon%2C+Arghavan+Moradi%2DDakhel%2C+Amin+Nikanjam%2C+Foutse+Khomh%2C+Michel+C+Desmarais%2C+and+Giuliano+Antoniol%2E+2025%2E+Bugs+in+large+language+models+generated+code%3A+An+empirical+study%2E+Empirical+Softwa)
43. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
44. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. [find](https://scholar.google.com/scholar?q=Gemma+Team%2C+Aishwarya+Kamath%2C+Johan+Ferret%2C+Shreya+Pathak%2C+Nino+Vieillard%2C+Ramona+Merhej%2C+Sarah+Perrin%2C+Tatiana+Matejovicova%2C+Alexandre+Ram%C3%A9%2C+Morgane+Rivi%C3%A8re%2C+and+1+others%2E+2025%2E+Gemma+3+technical+rep)
45. Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022. Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models. In Chi conference on human factors in computing systems extended abstracts, pages 1–7. [find](https://scholar.google.com/scholar?q=Priyan+Vaithilingam%2C+Tianyi+Zhang%2C+and+Elena+L+Glassman%2E+2022%2E+Expectation+vs%2E+experience%3A+Evaluating+the+usability+of+code+generation+tools+powered+by+large+language+models%2E+In+Chi+conference+on+huma)
46. Nalin Wadhwa, Jui Pradhan, Atharv Sonwane, Surya Prakash Sahu, Nagarajan Natarajan, Aditya Kanade, Suresh Parthasarathy, and Sriram Rajamani. 2024. Core: Resolving code quality issues using llms. Proceedings of the ACM on Software Engineering, 1(FSE):789–811. [find](https://scholar.google.com/scholar?q=Nalin+Wadhwa%2C+Jui+Pradhan%2C+Atharv+Sonwane%2C+Surya+Prakash+Sahu%2C+Nagarajan+Natarajan%2C+Aditya+Kanade%2C+Suresh+Parthasarathy%2C+and+Sriram+Rajamani%2E+2024%2E+Core%3A+Resolving+code+quality+issues+using+llms%2E+Proc)
47. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025b. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
48. Tong Ye, Yangkai Du, Tengfei Ma, Lingfei Wu, Xuhong Zhang, Shouling Ji, and Wenhai Wang. 2024. Uncovering llm-generated code: A zero-shot synthetic code detector via code rewriting. arXiv preprint arXiv:2405.16133. — [arXiv:2405.16133](https://arxiv.org/abs/2405.16133) [find](https://scholar.google.com/scholar?q=Tong+Ye%2C+Yangkai+Du%2C+Tengfei+Ma%2C+Lingfei+Wu%2C+Xuhong+Zhang%2C+Shouling+Ji%2C+and+Wenhai+Wang%2E+2024%2E+Uncovering+llm%2Dgenerated+code%3A+A+zero%2Dshot+synthetic+code+detector+via+code+rewriting%2E+arXiv+preprint+arX)
49. Kunhao Zheng, Juliette Decugis, Jonas Gehring, Taco Cohen, Benjamin Negrevergne, and Gabriel Synnaeve. 2024. What makes large language models reason in (multi-turn) code generation? arXiv preprint arXiv:2410.08105. — [arXiv:2410.08105](https://arxiv.org/abs/2410.08105) [find](https://scholar.google.com/scholar?q=Kunhao+Zheng%2C+Juliette+Decugis%2C+Jonas+Gehring%2C+Taco+Cohen%2C+Benjamin+Negrevergne%2C+and+Gabriel+Synnaeve%2E+2024%2E+What+makes+large+language+models+reason+in+%28multi%2Dturn%29+code+generation%3F+arXiv+preprint+arX)
50. Yao Wan, Guanghua Wan, Shijie Zhang, Hongyu Zhang, Pan Zhou, Hai Jin, and Lichao Sun. 2024. Does your neural code completion model use my code? a membership inference approach. arXiv preprint arXiv:2404.14296. — [arXiv:2404.14296](https://arxiv.org/abs/2404.14296) [find](https://scholar.google.com/scholar?q=Yao+Wan%2C+Guanghua+Wan%2C+Shijie+Zhang%2C+Hongyu+Zhang%2C+Pan+Zhou%2C+Hai+Jin%2C+and+Lichao+Sun%2E+2024%2E+Does+your+neural+code+completion+model+use+my+code%3F+a+membership+inference+approach%2E+arXiv+preprint+arXiv%3A24)
51. Li Zhong and Zilong Wang. 2024. Can llm replace stack overflow? a study on robustness and reliability of large language model code generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21841–21849. [find](https://scholar.google.com/scholar?q=Li+Zhong+and+Zilong+Wang%2E+2024%2E+Can+llm+replace+stack+overflow%3F+a+study+on+robustness+and+reliability+of+large+language+model+code+generation%2E+In+Proceedings+of+the+AAAI+Conference+on+Artificial+Intel)
52. Yanlin Wang, Tianyue Jiang, Mingwei Liu, Jiachi Chen, and Zibin Zheng. 2024. Beyond functional correctness: Investigating coding style inconsistencies in large language models. arXiv preprint arXiv:2407.00456. — [arXiv:2407.00456](https://arxiv.org/abs/2407.00456) [find](https://scholar.google.com/scholar?q=Yanlin+Wang%2C+Tianyue+Jiang%2C+Mingwei+Liu%2C+Jiachi+Chen%2C+and+Zibin+Zheng%2E+2024%2E+Beyond+functional+correctness%3A+Investigating+coding+style+inconsistencies+in+large+language+models%2E+arXiv+preprint+arXiv%3A24)
53. Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2022. Productivity assessment of neural code completion. In Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, pages 21–29. [find](https://scholar.google.com/scholar?q=Albert+Ziegler%2C+Eirini+Kalliamvakou%2C+X+Alice+Li%2C+Andrew+Rice%2C+Devon+Rifkin%2C+Shawn+Simister%2C+Ganesh+Sittampalam%2C+and+Edward+Aftandilian%2E+2022%2E+Productivity+assessment+of+neural+code+completion%2E+In+Proc)
54. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+CH+Hoi%2E+2021%2E+Codet5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2109%2E00859%2E)
55. Jinwei Xu, He Zhang, Yanjin Yang, Zeru Cheng, Jun Lyu, Bohan Liu, Xin Zhou, Lanxin Yang, Alberto Bacchelli, Yin Kia Chiam, and 1 others. 2024a. Investigating efficacy of perplexity in detecting llm-generated code. arXiv preprint arXiv:2412.16525. — [arXiv:2412.16525](https://arxiv.org/abs/2412.16525) [find](https://scholar.google.com/scholar?q=Jinwei+Xu%2C+He+Zhang%2C+Yanjin+Yang%2C+Zeru+Cheng%2C+Jun+Lyu%2C+Bohan+Liu%2C+Xin+Zhou%2C+Lanxin+Yang%2C+Alberto+Bacchelli%2C+Yin+Kia+Chiam%2C+and+1+others%2E+2024a%2E+Investigating+efficacy+of+perplexity+in+detecting+llm%2Dge)
56. Weiwei Xu, Kai Gao, Hao He, and Minghui Zhou. 2024b. Licoeval: Evaluating llms on license compliance in code generation. arXiv preprint arXiv:2408.02487. — [arXiv:2408.02487](https://arxiv.org/abs/2408.02487) [find](https://scholar.google.com/scholar?q=Weiwei+Xu%2C+Kai+Gao%2C+Hao+He%2C+and+Minghui+Zhou%2E+2024b%2E+Licoeval%3A+Evaluating+llms+on+license+compliance+in+code+generation%2E+arXiv+preprint+arXiv%3A2408%2E02487%2E+%E2%80%94+%5BarXiv%3A2408%2E02487%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F240)
57. Xiaodan Xu, Chao Ni, Xinrong Guo, Shaoxuan Liu, Xiaoya Wang, Kui Liu, and Xiaohu Yang. 2024c. Distinguishing llm-generated from human-written code by contrastive learning. ACM Transactions on Software Engineering and Methodology. [find](https://scholar.google.com/scholar?q=Xiaodan+Xu%2C+Chao+Ni%2C+Xinrong+Guo%2C+Shaoxuan+Liu%2C+Xiaoya+Wang%2C+Kui+Liu%2C+and+Xiaohu+Yang%2E+2024c%2E+Distinguishing+llm%2Dgenerated+from+human%2Dwritten+code+by+contrastive+learning%2E+ACM+Transactions+on+Software)
58. Dataset Human-Written Solutions We retrieved accepted Python and C/C++ code submissions from the Code4Bench database using SQL queries, as shown in Figure 8. For each Codeforces problem, we retain only a single human-written submission and then use BeautifulSoup to convert the problem description fr… [find](https://scholar.google.com/scholar?q=Dataset+Human%2DWritten+Solutions+We+retrieved+accepted+Python+and+C%2FC%2B%2B+code+submissions+from+the+Code4Bench+database+using+SQL+queries%2C+as+shown+in+Figure+8%2E+For+each+Codeforces+problem%2C+we+retain+onl)

*Updated 2 November 2026*