---
title: "A Study of LLMs' Preferences for Libraries and Programming"
description: "The paper examines a rarely evaluated aspect of LLM code generation: which libraries and… 85 references, cited by 2 papers here."
authors: ["Lukas Twist", "Mark Harman", "Don Syme", "Joost Noppen", "Helen Yannakoudakis", "Detlef Nauck", "Jie M. Zhang"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.15/"
doi: "10.18653/v1/2026.findings-acl.15"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/twist-2026-study-llms-preferences-libraries-programming/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/twist-2026-study-llms-preferences-libraries-programming/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/) / A Study of LLMs' Preferences for Libraries and Programming Languages

# A Study of LLMs' Preferences for Libraries and Programming Languages

**Authors:** Lukas Twist, Mark Harman, Don Syme, Joost Noppen, [Helen Yannakoudakis](https://md.research.elamur.ai/free/en/authors/helen-yannakoudakis/), Detlef Nauck and [Jie M. Zhang](https://md.research.elamur.ai/free/en/authors/jie-m-zhang/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.15/) · [PDF](https://aclanthology.org/2026.findings-acl.15.pdf) · [DOI 10.18653/v1/2026.findings-acl.15](https://doi.org/10.18653/v1/2026.findings-acl.15) · [OpenAlex](https://openalex.org/W4416840239)

**Citations:** 3 citations (OpenAlex, 2026-10-14) · 2 in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper examines a rarely evaluated aspect of LLM code generation: which libraries and programming languages a model picks when the prompt leaves that choice open. The authors test eight production LLMs, both open and closed, general-purpose and code-specialised, using default API settings and no system prompt. For libraries, they use 525 Python tasks from BigCodeBench whose descriptions do not name the required libraries. They also write five open-ended project initialisation prompts and collect 100 generations per model for each. Language preferences are studied with similar benchmark and project-level setups. The authors also check whether the technologies a model recommends in prose match the ones it actually uses in code.

All models favour established, popular tools over newer, fast-growing alternatives. They often import libraries such as NumPy when the task does not call for them. They default heavily to Python, even for performance-critical work. This matters for security, efficiency and open-source diversity, because LLM defaults can reinforce dominant ecosystems over time.

## Key points
- Models studied: GPT-4o-mini, GPT-3.5-turbo, Claude-3.5 Sonnet, Claude-3.5 Haiku, Llama-3.2-3B, Mistral-7B, Qwen-2.5-Coder and DeepSeek-LLM.
- The library benchmark is a filtered subset of BigCodeBench, reduced from 1,140 to 525 tasks covering 34 ground-truth libraries, with three responses per task.
- NumPy usage disagrees with the ground-truth solutions in up to 45% of tasks.
- For high-performance project initialisation tasks, models select Python more often than any other language, doing so 58% of the time, while Rust is not picked even once.
- The libraries and languages models recommend in natural language are a poor predictor of what they actually use in code. Code and data are released at github.com/itsluketwist/llm-code-bias.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 88 works · 19,432 citations
- [King's College London](https://md.research.elamur.ai/free/en/universities/king-s-college-london/) — University, London, United Kingdom · 42 works · 1,617 citations

Also on the byline: [BT Research](https://ror.org/03308db07).

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck and Jie M. Zhang. 2026. A Study of LLMs' Preferences for Libraries and Programming Languages. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{twist2026study,
  title = {{A Study of LLMs' Preferences for Libraries and Programming Languages}},
  author = {Lukas Twist, Mark Harman, Don Syme, Joost Noppen, Helen Yannakoudakis, Detlef Nauck and Jie M. Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.15},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.15}
}
```

- [← previous Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Models](https://md.research.elamur.ai/free/en/papers/xue-2026-why-supervised-fine-tuning-fails/)
- [next → ARCQuant: Boosting NVFP4 Quantization with Augmented Residual Channels for LLMs](https://md.research.elamur.ai/free/en/papers/meng-2026-arcquant-boosting-nvfp4-quantization-augmented/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/3/)

## References

85 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mehmet Akhoroz and Caglar Yildirim. 2025. Conversational AI as a Coding Assistant: Understanding Programmers’ Interactions with and Expectations from Large Language Models for Coding. Preprint, arXiv:2503.16508. — [arXiv:2503.16508](https://arxiv.org/abs/2503.16508) [find](https://scholar.google.com/scholar?q=Mehmet+Akhoroz+and+Caglar+Yildirim%2E+2025%2E+Conversational+AI+as+a+Coding+Assistant%3A+Understanding+Programmers%E2%80%99+Interactions+with+and+Expectations+from+Large+Language+Models+for+Coding%2E+Preprint%2C+arXiv%3A)
2. Andrew Peng, Michael Wu, John Allard, Logan Kilpatrick, and Steven Heidel. 2023. [find](https://scholar.google.com/scholar?q=Andrew+Peng%2C+Michael+Wu%2C+John+Allard%2C+Logan+Kilpatrick%2C+and+Steven+Heidel%2E+2023%2E)
3. GPT-3.5 Turbo fine-tuning and API updates. [https://openai.com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/](https://openai.com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/). [↗](https://openai.com/index/gpt-3-5-turbo-fine-tuning-and-api-updates/)
4. Anthropic. 2024. Claude 3 Model Card. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2024%2E+Claude+3+Model+Card%2E)
5. Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan… [find](https://scholar.google.com/scholar?q=Ben+Athiwaratkun%2C+Sanjay+Krishna+Gouda%2C+Zijian+Wang%2C+Xiaopeng+Li%2C+Yuchen+Tian%2C+Ming+Tan%2C+Wasi+Uddin+Ahmad%2C+Shiqi+Wang%2C+Qing+Sun%2C+Mingyue+Shang%2C+Sujan+Kumar+Gonugondla%2C+Hantian+Ding%2C+Varun+Kumar%2C+Natha)
6. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). Preprint, arXiv:2108.07732.
7. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623, Virtual Event Canada. ACM. [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+Dangers+of+Stochastic+Parrots%3A+Can+Language+Models+Be+Too+Big%3F+In+Proceedings+of+the+2021+ACM+Conference)
8. Rishi Bommasani, Kevin Klyman, Shayne Longpre, Sayash Kapoor, Nestor Maslej, Betty Xiong, Daniel Zhang, and Percy Liang. 2023. The Foundation Model Transparency Index. Preprint, arXiv:2310.12941. [find](https://scholar.google.com/scholar?q=Rishi+Bommasani%2C+Kevin+Klyman%2C+Shayne+Longpre%2C+Sayash+Kapoor%2C+Nestor+Maslej%2C+Betty+Xiong%2C+Daniel+Zhang%2C+and+Percy+Liang%2E+2023%2E+The+Foundation+Model+Transparency+Index%2E+Preprint%2C+arXiv%3A2310%2E12941%2E)
9. William Bugden and Ayman Alahmar. 2022. The Safety and Performance of Prominent Programming Languages. International Journal of Software Engineering and Knowledge Engineering, 32(05):713–744. [find](https://scholar.google.com/scholar?q=William+Bugden+and+Ayman+Alahmar%2E+2022%2E+The+Safety+and+Performance+of+Prominent+Programming+Languages%2E+International+Journal+of+Software+Engineering+and+Knowledge+Engineering%2C+32%2805%29%3A713%E2%80%93744%2E)
10. Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, Wei Ye, and Shikun Zhang. 2024a. A Survey on Evaluating Large Language Models in Code Generation Tasks. Journal of computer science and technology. [find](https://scholar.google.com/scholar?q=Liguo+Chen%2C+Qi+Guo%2C+Hongrui+Jia%2C+Zhengran+Zeng%2C+Xin+Wang%2C+Yijiang+Xu%2C+Jian+Wu%2C+Yidong+Wang%2C+Qing+Gao%2C+Jindong+Wang%2C+Wei+Ye%2C+and+Shikun+Zhang%2E+2024a%2E+A+Survey+on+Evaluating+Large+Language+Models+in+Cod)
11. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
12. Yujia Chen, Cuiyun Gao, Muyijie Zhu, Qing Liao, Yong Wang, and Guoai Xu. 2024b. [find](https://scholar.google.com/scholar?q=Yujia+Chen%2C+Cuiyun+Gao%2C+Muyijie+Zhu%2C+Qing+Liao%2C+Yong+Wang%2C+and+Guoai+Xu%2E+2024b%2E)
13. APIGen: Generative API Method Recommendation. In 2024 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 171–182. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=APIGen%3A+Generative+API+Method+Recommendation%2E+In+2024+IEEE+International+Conference+on+Software+Analysis%2C+Evolution+and+Reengineering+%28SANER%29%2C+pages+171%E2%80%93182%2E+IEEE+Computer+Society%2E)
14. Jishnu Ray Chowdhury and Cornelia Caragea. 2025. Zero-Shot Verification-guided Chain of Thoughts. Preprint, arXiv:2501.13122. — [arXiv:2501.13122](https://arxiv.org/abs/2501.13122) [find](https://scholar.google.com/scholar?q=Jishnu+Ray+Chowdhury+and+Cornelia+Caragea%2E+2025%2E+Zero%2DShot+Verification%2Dguided+Chain+of+Thoughts%2E+Preprint%2C+arXiv%3A2501%2E13122%2E+%E2%80%94+%5BarXiv%3A2501%2E13122%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2501%2E13122%29)
15. Manuel Costanzo, Enzo Rucci, Marcelo Naiouf, and Armando De Giusti. 2021. Performance vs Programming Effort between Rust and C on Multicore Architectures: Case Study in N-Body. In 2021 XLVII Latin American Computing Conference (CLEI), pages 1– 10. [find](https://scholar.google.com/scholar?q=Manuel+Costanzo%2C+Enzo+Rucci%2C+Marcelo+Naiouf%2C+and+Armando+De+Giusti%2E+2021%2E+Performance+vs+Programming+Effort+between+Rust+and+C+on+Multicore+Architectures%3A+Case+Study+in+N%2DBody%2E+In+2021+XLVII+Latin+Ame)
16. DeepMind. 2024. Competitive programming with AlphaCode. [https://deepmind.google/discover/blog/competitive-programming-with-alphacode/](https://deepmind.google/discover/blog/competitive-programming-with-alphacode/). [↗](https://deepmind.google/discover/blog/competitive-programming-with-alphacode/)
17. DeepSeek-AI, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, and 68 others. 2024. DeepSeek LLM: Scaling Open-Source Language Models wit… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Xiao+Bi%2C+Deli+Chen%2C+Guanting+Chen%2C+Shanhuang+Chen%2C+Damai+Dai%2C+Chengqi+Deng%2C+Honghui+Ding%2C+Kai+Dong%2C+Qiushi+Du%2C+Zhe+Fu%2C+Huazuo+Gao%2C+Kaige+Gao%2C+Wenjun+Gao%2C+Ruiqi+Ge%2C+Kang+Guan%2C+Daya+Guo%2C+Ji)
18. GitHub Staff. 2024. Octoverse: AI leads Python to top language as the number of global developers surges. [find](https://scholar.google.com/scholar?q=GitHub+Staff%2E+2024%2E+Octoverse%3A+AI+leads+Python+to+top+language+as+the+number+of+global+developers+surges%2E)
19. Sam Gross. 2023. PEP 703 – Making the Global Interpreter Lock Optional in CPython \| peps.python.org. [https://peps.python.org/pep-0703/](https://peps.python.org/pep-0703/). [↗](https://peps.python.org/pep-0703/)
20. Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi, Percy Liang, and Tatsunori Hashimoto. 2025. Auditing prompt caching in language model APIs. In Forty-second International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Chenchen+Gu%2C+Xiang+Lisa+Li%2C+Rohith+Kuditipudi%2C+Percy+Liang%2C+and+Tatsunori+Hashimoto%2E+2025%2E+Auditing+prompt+caching+in+language+model+APIs%2E+In+Forty%2Dsecond+International+Conference+on+Machine+Learning%2E)
21. Yufei Guo, Muzhe Guo, Juntao Su, Zhou Yang, Mengqiu Zhu, Hongfei Li, Mengyang Qiu, and Shuo Shuo Liu. 2024. [Bias in Large Language Models: Origin, Evaluation, and Mitigation](https://md.research.elamur.ai/free/en/papers/guo-2024-bias-large-language-models-origin/). Preprint, arXiv:2411.10915.
22. Huizi Hao, Kazi Amit Hasan, Hong Qin, Marcos Macedo, Yuan Tian, Steven H. H. Ding, and Ahmed E. Hassan. 2024. An Empirical Study on Developers Shared Conversations with ChatGPT in GitHub Pull Requests and Issues. Empirical Software Engineering, 29(6):150. [find](https://scholar.google.com/scholar?q=Huizi+Hao%2C+Kazi+Amit+Hasan%2C+Hong+Qin%2C+Marcos+Macedo%2C+Yuan+Tian%2C+Steven+H%2E+H%2E+Ding%2C+and+Ahmed+E%2E+Hassan%2E+2024%2E+An+Empirical+Study+on+Developers+Shared+Conversations+with+ChatGPT+in+GitHub+Pull+Requests)
23. R. Maria del Rio-Chanona, Nadzeya Laurentsyeva, and Johannes Wachs. 2024. Large language models reduce public knowledge sharing on online Q&A platforms. PNAS Nexus, 3(9). [find](https://scholar.google.com/scholar?q=R%2E+Maria+del+Rio%2DChanona%2C+Nadzeya+Laurentsyeva%2C+and+Johannes+Wachs%2E+2024%2E+Large+language+models+reduce+public+knowledge+sharing+on+online+Q%26A+platforms%2E+PNAS+Nexus%2C+3%289%29%2E)
24. Yiyang Hao, Ge Li, Yongqiang Liu, Xiaowei Miao, He Zong, Siyuan Jiang, Yang Liu, and He Wei. 2022. AixBench: A Code Generation Benchmark Dataset. Preprint, arXiv:2206.13179. — [arXiv:2206.13179](https://arxiv.org/abs/2206.13179) [find](https://scholar.google.com/scholar?q=Yiyang+Hao%2C+Ge+Li%2C+Yongqiang+Liu%2C+Xiaowei+Miao%2C+He+Zong%2C+Siyuan+Jiang%2C+Yang+Liu%2C+and+He+Wei%2E+2022%2E+AixBench%3A+A+Code+Generation+Benchmark+Dataset%2E+Preprint%2C+arXiv%3A2206%2E13179%2E+%E2%80%94+%5BarXiv%3A2206%2E13179%5D%28https)
25. Benedetta Donato, Leonardo Mariani, Daniela Micucci, and Oliviero Riganelli. 2025. Studying How Configurations Impact Code Generation in LLMs: The Case of ChatGPT. In The Proceedings of the 33rd IEEE/ACM International Conference on Program Comprehension. arXiv. [find](https://scholar.google.com/scholar?q=Benedetta+Donato%2C+Leonardo+Mariani%2C+Daniela+Micucci%2C+and+Oliviero+Riganelli%2E+2025%2E+Studying+How+Configurations+Impact+Code+Generation+in+LLMs%3A+The+Case+of+ChatGPT%2E+In+The+Proceedings+of+the+33rd+IEEE%2F)
26. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring Coding Challenge Competence With APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). In Proceedings of the Neural Information Processing Systems Track on Datasets and Benc…
27. Hanpei Fang, Sijie Tao, Nuo Chen, Kai-Xin Chang, and Tetsuya Sakai. 2025. Do Large Language Models Favor Recent Content? A Study on Recency Bias in LLM-Based Reranking. Preprint, arXiv:2509.11353. — [arXiv:2509.11353](https://arxiv.org/abs/2509.11353) [find](https://scholar.google.com/scholar?q=Hanpei+Fang%2C+Sijie+Tao%2C+Nuo+Chen%2C+Kai%2DXin+Chang%2C+and+Tetsuya+Sakai%2E+2025%2E+Do+Large+Language+Models+Favor+Recent+Content%3F+A+Study+on+Recency+Bias+in+LLM%2DBased+Reranking%2E+Preprint%2C+arXiv%3A2509%2E11353%2E+%E2%80%94+%5B)
28. Carlo A. Furia, Richard Torkar, and Robert Feldt. 2024. Towards Causal Analysis of Empirical Software Engineering Data: The Impact of Programming Languages on Coding Competitions. ACM Transactions on Software Engineering and Methodology, 33(1):1– 35. [find](https://scholar.google.com/scholar?q=Carlo+A%2E+Furia%2C+Richard+Torkar%2C+and+Robert+Feldt%2E+2024%2E+Towards+Causal+Analysis+of+Empirical+Software+Engineering+Data%3A+The+Impact+of+Programming+Languages+on+Coding+Competitions%2E+ACM+Transactions+on+)
29. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024. [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/). Computational Linguistics, 50(3):1097– 1179.
30. Yulia Gavrilova. 2023. Pros and Cons of Python. [https://serokell.io/blog/python-pros-and-cons](https://serokell.io/blog/python-pros-and-cons). [↗](https://serokell.io/blog/python-pros-and-cons)
31. Daniel M. German, Massimiliano Di Penta, Yann-Gael Gueheneuc, and Giuliano Antoniol. 2009. Code siblings: Technical and legal implications of copying code between applications. In 2009 6th IEEE International Working Conference on Mining Software Repositories, pages 81–90. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+German%2C+Massimiliano+Di+Penta%2C+Yann%2DGael+Gueheneuc%2C+and+Giuliano+Antoniol%2E+2009%2E+Code+siblings%3A+Technical+and+legal+implications+of+copying+code+between+applications%2E+In+2009+6th+IEEE+Intern)
32. Dong Huang, Jie M. Zhang, Qingwen Bu, Xiaofei Xie, Junjie Chen, and Heming Cui. 2024. Bias Testing and Mitigation in LLM-based Code Generation. ACM Transactions on Software Engineering and Methodology. hugovk. 2025. Top PyPI Packages. [https://hugovk.github.io/top-pypi-packages/](https://hugovk.github.io/top-pypi-packages/). [↗](https://hugovk.github.io/top-pypi-packages/)
33. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. [Qwen2.5-Coder Technical Report](https://md.research.elamur.ai/free/en/papers/hui-2024-qwen2-5-coder-technical-report/). Preprint, arXiv:2409.12186.
34. Paul Jansen. 2025. TIOBE Index. [find](https://scholar.google.com/scholar?q=Paul+Jansen%2E+2025%2E+TIOBE+Index%2E)
35. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
36. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/). ACM Transactions on Software Engineering and Methodology.
37. Dawid Karczewski. 2021. Python vs C++: Technology Comparison. [https://www.ideamotive.co/blog/python-vs-cpp-technology-comparison](https://www.ideamotive.co/blog/python-vs-cpp-technology-comparison). [↗](https://www.ideamotive.co/blog/python-vs-cpp-technology-comparison)
38. Maurice G. Kendall and Jean Dickinson Gibbons. 1990. Rank Correlation Methods, 5th ed edition. Oxford University Press, New York, NY. [find](https://scholar.google.com/scholar?q=Maurice+G%2E+Kendall+and+Jean+Dickinson+Gibbons%2E+1990%2E+Rank+Correlation+Methods%2C+5th+ed+edition%2E+Oxford+University+Press%2C+New+York%2C+NY%2E)
39. Anjali Khurana, Hari Subramonyam, and Parmit K. Chilana. 2024. Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 288– 303. [find](https://scholar.google.com/scholar?q=Anjali+Khurana%2C+Hari+Subramonyam%2C+and+Parmit+K%2E+Chilana%2E+2024%2E+Why+and+When+LLM%2DBased+Assistants+Can+Go+Wrong%3A+Investigating+the+Effectiveness+of+Prompt%2DBased+Interactions+for+Software+Help%2DSeeking%2E+I)
40. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. [Large language models are zero-shot reasoners](https://md.research.elamur.ai/free/en/papers/kojima-2022-large-language-models-zero-shot/). In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, pages 22199–22213, Red Hook, NY, USA. Curran Associates Inc.
41. Adrian Kuhn and Robert DeLine. 2012. On designing better tools for learning APIs . In 2012 4th International Workshop on Search-driven Development: Users, Infrastructure, Tools, and Evaluation (SUITE 2012), pages 27–30, Los Alamitos, CA, USA. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=Adrian+Kuhn+and+Robert+DeLine%2E+2012%2E+On+designing+better+tools+for+learning+APIs+%2E+In+2012+4th+International+Workshop+on+Search%2Ddriven+Development%3A+Users%2C+Infrastructure%2C+Tools%2C+and+Evaluation+%28SUITE+)
42. Enrique Larios-Vargas, Maurício Aniche, Christoph Treude, Magiel Bruntink, and Georgios Gousios. 2020. Selecting third-party libraries: The practitioners’ perspective. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Softwar… [find](https://scholar.google.com/scholar?q=Enrique+Larios%2DVargas%2C+Maur%C3%ADcio+Aniche%2C+Christoph+Treude%2C+Magiel+Bruntink%2C+and+Georgios+Gousios%2E+2020%2E+Selecting+third%2Dparty+libraries%3A+The+practitioners%E2%80%99+perspective%2E+In+Proceedings+of+the+28th+ACM+J)
43. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, and 7 o… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
44. Yan Liu, Xiaokang Chen, Yan Gao, Zhe Su, Fengji Zhang, Daoguang Zan, Jian-Guang Lou, Pin-Yu Chen, and Tsung-Yi Ho. 2023b. Uncovering and Quantifying Social Biases in Code Generation. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2… [find](https://scholar.google.com/scholar?q=Yan+Liu%2C+Xiaokang+Chen%2C+Yan+Gao%2C+Zhe+Su%2C+Fengji+Zhang%2C+Daoguang+Zan%2C+Jian%2DGuang+Lou%2C+Pin%2DYu+Chen%2C+and+Tsung%2DYi+Ho%2E+2023b%2E+Uncovering+and+Quantifying+Social+Biases+in+Code+Generation%2E+In+Advances+in+Ne)
45. Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. 2024. Quantifying Variance in Evaluation Benchmarks. Preprint, arXiv:2406.10229. [find](https://scholar.google.com/scholar?q=Lovish+Madaan%2C+Aaditya+K%2E+Singh%2C+Rylan+Schaeffer%2C+Andrew+Poulton%2C+Sanmi+Koyejo%2C+Pontus+Stenetorp%2C+Sharan+Narang%2C+and+Dieuwke+Hupkes%2E+2024%2E+Quantifying+Variance+in+Evaluation+Benchmarks%2E+Preprint%2C+arXi)
46. Vahid Majdinasab, Amin Nikanjam, and Foutse Khomh. 2025. Trained Without My Consent: Detecting Code Inclusion In Language Models Trained on Code. ACM Transactions on Software Engineering and Methodology, 34(4):1–46. [find](https://scholar.google.com/scholar?q=Vahid+Majdinasab%2C+Amin+Nikanjam%2C+and+Foutse+Khomh%2E+2025%2E+Trained+Without+My+Consent%3A+Detecting+Code+Inclusion+In+Language+Models+Trained+on+Code%2E+ACM+Transactions+on+Software+Engineering+and+Methodolo)
47. Markdown Guide. 2025. Extended Syntax \| Markdown Guide. [https://www.markdownguide.org/extended-syntax/](https://www.markdownguide.org/extended-syntax/). [↗](https://www.markdownguide.org/extended-syntax/)
48. Alexandre Matton, Tom Sherborne, Dennis Aumiller, Elena Tommasone, Milad Alizadeh, Jingyi He, Raymond Ma, Maxime Voisin, Ellen Gilsenan-McMahon, and Matthias Gallé. 2024. On Leakage of Code Generation Evaluation Datasets. In Findings of the Association for Computational Linguistics: EMNLP 2024, page… [find](https://scholar.google.com/scholar?q=Alexandre+Matton%2C+Tom+Sherborne%2C+Dennis+Aumiller%2C+Elena+Tommasone%2C+Milad+Alizadeh%2C+Jingyi+He%2C+Raymond+Ma%2C+Maxime+Voisin%2C+Ellen+Gilsenan%2DMcMahon%2C+and+Matthias+Gall%C3%A9%2E+2024%2E+On+Leakage+of+Code+Generation)
49. Meta. 2025. Llama 3.2 \| Model Cards and Prompt formats. [https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/). [↗](https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_2/)
50. Norman Mu, Jonathan Lu, Michael Lavery, and David Wagner. 2024. A closer look at system message robustness. In Neurips Safe Generative AI Workshop 2024. [find](https://scholar.google.com/scholar?q=Norman+Mu%2C+Jonathan+Lu%2C+Michael+Lavery%2C+and+David+Wagner%2E+2024%2E+A+closer+look+at+system+message+robustness%2E+In+Neurips+Safe+Generative+AI+Workshop+2024%2E)
51. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
52. Lin Ling, Fazle Rabbi, Song Wang, and Jinqiu Yang. 2025. Bias Unveiled: Investigating Social Bias in LLM-Generated Code. Proceedings of the AAAI Conference on Artificial Intelligence, 39(26):27491– 27499. [find](https://scholar.google.com/scholar?q=Lin+Ling%2C+Fazle+Rabbi%2C+Song+Wang%2C+and+Jinqiu+Yang%2E+2025%2E+Bias+Unveiled%3A+Investigating+Social+Bias+in+LLM%2DGenerated+Code%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+39%2826%29%3A27491%E2%80%93+27)
53. Humza Naveed, Asad Ullah Khan, Shi Qiu, Muhammad Saqib, Saeed Anwar, Muhammad Usman, Naveed Akhtar, Nick Barnes, and Ajmal Mian. 2024. [A Comprehensive Overview of Large Language Models](https://md.research.elamur.ai/free/en/papers/naveed-2023-comprehensive-overview-large-language-models/). ACM Transactions on Intelligent Systems and Technology, 16(5):1–72.
54. Mingwei Liu, Tianyong Yang, Yiling Lou, Xueying Du, Ying Wang, and Xin Peng. 2023a. CodeGen4Libs: A Two-Stage Approach for Library-Oriented Code Generation. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 434–445, Luxembourg, Luxembourg. IEEE. [find](https://scholar.google.com/scholar?q=Mingwei+Liu%2C+Tianyong+Yang%2C+Yiling+Lou%2C+Xueying+Du%2C+Ying+Wang%2C+and+Xin+Peng%2E+2023a%2E+CodeGen4Libs%3A+A+Two%2DStage+Approach+for+Library%2DOriented+Code+Generation%2E+In+2023+38th+IEEE%2FACM+International+Confere)
55. Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q. Feldman. 2024. How Beginning Programmers and Code LLMs (Mis)read Each Other. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1–26. [find](https://scholar.google.com/scholar?q=Sydney+Nguyen%2C+Hannah+McLean+Babe%2C+Yangtian+Zi%2C+Arjun+Guha%2C+Carolyn+Jane+Anderson%2C+and+Molly+Q%2E+Feldman%2E+2024%2E+How+Beginning+Programmers+and+Code+LLMs+%28Mis%29read+Each+Other%2E+In+Proceedings+of+the+CHI+C)
56. Muhammed Nihal. 2024. The Race to Zero Latency: How to Optimize Code for High-Frequency Trading Quant Firms. [find](https://scholar.google.com/scholar?q=Muhammed+Nihal%2E+2024%2E+The+Race+to+Zero+Latency%3A+How+to+Optimize+Code+for+High%2DFrequency+Trading+Quant+Firms%2E)
57. Mbithe Nzomo. 2025. Absolute vs Relative Imports in Python – Real Python. [https://realpython.com/absolute-vs-relative-python-imports/](https://realpython.com/absolute-vs-relative-python-imports/). [↗](https://realpython.com/absolute-vs-relative-python-imports/)
58. OpenAI. 2025. Models [https://platform.openai.com](https://platform.openai.com). - OpenAI API. [↗](https://platform.openai.com)
59. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — [GPT-4 Technical Report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/)
60. Arkil Patel, Siva Reddy, Dzmitry Bahdanau, and Pradeep Dasigi. 2024. Evaluating In-Context Learning of Libraries for Code Generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape… [find](https://scholar.google.com/scholar?q=Arkil+Patel%2C+Siva+Reddy%2C+Dzmitry+Bahdanau%2C+and+Pradeep+Dasigi%2E+2024%2E+Evaluating+In%2DContext+Learning+of+Libraries+for+Code+Generation%2E+In+Proceedings+of+the+2024+Conference+of+the+North+American+Chapte)
61. Matthew Renze and Erhan Guven. 2024. [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://md.research.elamur.ai/free/en/papers/renze-2024-effect-sampling-temperature-problem-solving/). In Findings of the Association for Computational Linguistics: {EMNLP} 2024, Miami, Florida, USA, November 12-16, 2024. arXiv.
62. June Sallou, Thomas Durieux, and Annibale Panichella. 2024. Breaking the Silence: The Threats of Using LLMs in Software Engineering. In Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results, ICSE-NIER’24, pages 102–106, New York, NY, U… [find](https://scholar.google.com/scholar?q=June+Sallou%2C+Thomas+Durieux%2C+and+Annibale+Panichella%2E+2024%2E+Breaking+the+Silence%3A+The+Threats+of+Using+LLMs+in+Software+Engineering%2E+In+Proceedings+of+the+2024+ACM%2FIEEE+44th+International+Conference+o)
63. Advait Sarkar and Ian Drosos. 2025. Vibe coding: Programming through conversation with artificial intelligence. Preprint, arXiv:2506.23253. — [arXiv:2506.23253](https://arxiv.org/abs/2506.23253) [find](https://scholar.google.com/scholar?q=Advait+Sarkar+and+Ian+Drosos%2E+2025%2E+Vibe+coding%3A+Programming+through+conversation+with+artificial+intelligence%2E+Preprint%2C+arXiv%3A2506%2E23253%2E+%E2%80%94+%5BarXiv%3A2506%2E23253%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2506%2E23253%29)
64. SciPy. 2025. Kendalltau — SciPy v1.16.2 Manual. [https://docs.scipy.org/doc/scipy/reference/generated/](https://docs.scipy.org/doc/scipy/reference/generated/) scipy.stats.kendalltau.html. [↗](https://docs.scipy.org/doc/scipy/reference/generated/)
65. Matthew Smith. 2025. AI Vibe Coding: Engineers’ Secret to Fast Development - IEEE Spectrum. [https://spectrum.ieee.org/vibe-coding](https://spectrum.ieee.org/vibe-coding). [↗](https://spectrum.ieee.org/vibe-coding)
66. Ian Somerville. 2016. Software Engineering, Global Edition. Pearson Education. [find](https://scholar.google.com/scholar?q=Ian+Somerville%2E+2016%2E+Software+Engineering%2C+Global+Edition%2E+Pearson+Education%2E)
67. Minaoar Hossain Tanzil, Gias Uddin, and Ann Barcomb. 2024. "How do people decide?": A Model for Software Library Selection. In Proceedings of the 2024 IEEE/ACM 17th International Conference on Cooperative and Human Aspects of Software Engineering, pages 1–12. [find](https://scholar.google.com/scholar?q=Minaoar+Hossain+Tanzil%2C+Gias+Uddin%2C+and+Ann+Barcomb%2E+2024%2E+%22How+do+people+decide%3F%22%3A+A+Model+for+Software+Library+Selection%2E+In+Proceedings+of+the+2024+IEEE%2FACM+17th+International+Conference+on+Coopera)
68. Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. [Gorilla: Large language model connected with massive apis](https://md.research.elamur.ai/free/en/papers/patil-2023-gorilla-large-language-model-connected/). In Advances in Neural Information Processing Systems, volume 37, pages 126544–126565. Curran Associates, Inc.
69. Debalina Ghosh Paul, Hong Zhu, and Ian Bayley. 2024. Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review. In 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), pages 87–94. IEEE Computer Society. [find](https://scholar.google.com/scholar?q=Debalina+Ghosh+Paul%2C+Hong+Zhu%2C+and+Ian+Bayley%2E+2024%2E+Benchmarks+and+Metrics+for+Evaluations+of+Code+Generation%3A+A+Critical+Review%2E+In+2024+IEEE+International+Conference+on+Artificial+Intelligence+Test)
70. Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is Temperature the Creativity Parameter of Large Language Models? In International Conference on Computational Creativity. arXiv. [find](https://scholar.google.com/scholar?q=Max+Peeperkorn%2C+Tom+Kouwenhoven%2C+Dan+Brown%2C+and+Anna+Jordanous%2E+2024%2E+Is+Temperature+the+Creativity+Parameter+of+Large+Language+Models%3F+In+International+Conference+on+Computational+Creativity%2E+arXiv%2E)
71. PyPI. 2025. PyPI · The Python Package Index. [https://pypi.org/](https://pypi.org/). [↗](https://pypi.org/)
72. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. [ToolLLM: Facilitating Large Language Models to Master 16000+ Real…](https://md.research.elamur.ai/free/en/papers/qin-2023-toolllm-facilitating-large-language-models/)
73. Sanka Rasnayaka, Guanlin Wang, Ridwan Shariffdeen, and Ganesh Neelakanta Iyer. 2024. An Empirical Study on Usage and Perceptions of LLMs in a Software Engineering Project. In Proceedings of the 1st International Workshop on Large Language Models for Code, LLM4Code ’24, pages 111–118, New York, NY, U… [find](https://scholar.google.com/scholar?q=Sanka+Rasnayaka%2C+Guanlin+Wang%2C+Ridwan+Shariffdeen%2C+and+Ganesh+Neelakanta+Iyer%2E+2024%2E+An+Empirical+Study+on+Usage+and+Perceptions+of+LLMs+in+a+Software+Engineering+Project%2E+In+Proceedings+of+the+1st+In)
74. Lukas Twist and Jie M. Zhang. 2026. A Study of Library Usage in Agent-Authored Pull Requests. In Proceedings of the 23rd International Conference on Mining Software Repositories, MSR ’26, Rio de Janeiro, Brazil. ACM. vinta. 2025. Awesome Python. python.com/. https://awesome- [find](https://scholar.google.com/scholar?q=Lukas+Twist+and+Jie+M%2E+Zhang%2E+2026%2E+A+Study+of+Library+Usage+in+Agent%2DAuthored+Pull+Requests%2E+In+Proceedings+of+the+23rd+International+Conference+on+Mining+Software+Repositories%2C+MSR+%E2%80%9926%2C+Rio+de+Janei)
75. Chaozheng Wang, Zongjie Li, Cuiyun Gao, Wenxuan Wang, Ting Peng, Hailiang Huang, Yuetang Deng, Shuai Wang, and Michael R. Lyu. 2024. Exploring Multi-Lingual Bias of Large Code Models in Code Generation. Preprint, arXiv:2404.19368. — [arXiv:2404.19368](https://arxiv.org/abs/2404.19368) [find](https://scholar.google.com/scholar?q=Chaozheng+Wang%2C+Zongjie+Li%2C+Cuiyun+Gao%2C+Wenxuan+Wang%2C+Ting+Peng%2C+Hailiang+Huang%2C+Yuetang+Deng%2C+Shuai+Wang%2C+and+Michael+R%2E+Lyu%2E+2024%2E+Exploring+Multi%2DLingual+Bias+of+Large+Code+Models+in+Code+Generatio)
76. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, p…
77. Ankit Yadav, Himanshu Beniwal, and Mayank Singh. 2024. PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17113–17126, Miami, Florida, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ankit+Yadav%2C+Himanshu+Beniwal%2C+and+Mayank+Singh%2E+2024%2E+PythonSaga%3A+Redefining+the+Benchmark+to+Evaluate+Code+Generating+LLMs%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2024%2C+)
78. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. [Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/). In Proceedings of the 15th International Conference on Mining Software Repositories, {MSR} 2018, Gothenburg, Sweden, May 28-29, 2018. arXiv.
79. Daoguang Zan, Bei Chen, Yongshun Gong, Junzhi Cao, Fengji Zhang, Bingchao Wu, Bei Guan, Yilong Yin, and Yongji Wang. 2023. Private-Library-Oriented Code Generation with Large Language Models. Preprint, arXiv:2307.15370. — [arXiv:2307.15370](https://arxiv.org/abs/2307.15370) [find](https://scholar.google.com/scholar?q=Daoguang+Zan%2C+Bei+Chen%2C+Yongshun+Gong%2C+Junzhi+Cao%2C+Fengji+Zhang%2C+Bingchao+Wu%2C+Bei+Guan%2C+Yilong+Yin%2C+and+Yongji+Wang%2E+2023%2E+Private%2DLibrary%2DOriented+Code+Generation+with+Large+Language+Models%2E+Preprint)
80. Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Qingshuang Bao, Weipeng Jiang, Qian Wang, Chao Shen, and Yang Liu. 2025. The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1… [find](https://scholar.google.com/scholar?q=Xiaoyu+Zhang%2C+Juan+Zhai%2C+Shiqing+Ma%2C+Qingshuang+Bao%2C+Weipeng+Jiang%2C+Qian+Wang%2C+Chao+Shen%2C+and+Yang+Liu%2E+2025%2E+The+Invisible+Hand%3A+Unveiling+Provider+Bias+in+Large+Language+Models+for+Code+Generation%2E+)
81. Li Zhong and Zilong Wang. 2024. Can LLM replace stack overflow? a study on robustness and reliability of large language model code generation. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intellige… [find](https://scholar.google.com/scholar?q=Li+Zhong+and+Zilong+Wang%2E+2024%2E+Can+LLM+replace+stack+overflow%3F+a+study+on+robustness+and+reliability+of+large+language+model+code+generation%2E+In+Proceedings+of+the+Thirty%2DEighth+AAAI+Conference+on+Ar)
82. Xin Zhou, Martin Weyssow, Ratnadira Widyasari, Ting Zhang, Junda He, Yunbo Lyu, Jianming Chang, Beiqi Zhang, Dan Huang, and David Lo. 2025. LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks. Preprint, arXiv:2502.06215. — [arXiv:2502.06215](https://arxiv.org/abs/2502.06215) [find](https://scholar.google.com/scholar?q=Xin+Zhou%2C+Martin+Weyssow%2C+Ratnadira+Widyasari%2C+Ting+Zhang%2C+Junda+He%2C+Yunbo+Lyu%2C+Jianming+Chang%2C+Beiqi+Zhang%2C+Dan+Huang%2C+and+David+Lo%2E+2025%2E+LessLeak%2DBench%3A+A+First+Investigation+of+Data+Leakage+in+LLM)
83. Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, Ermo Hua, Xingtai Lv, Ning Ding, Zhouhan Lin, Zilong Zheng, and Bowen Zhou. 2025. How to Synthesize Text Data without Model Collapse? Preprint, arXiv:2412.14689. — [arXiv:2412.14689](https://arxiv.org/abs/2412.14689) [find](https://scholar.google.com/scholar?q=Xuekai+Zhu%2C+Daixuan+Cheng%2C+Hengli+Li%2C+Kaiyan+Zhang%2C+Ermo+Hua%2C+Xingtai+Lv%2C+Ning+Ding%2C+Zhouhan+Lin%2C+Zilong+Zheng%2C+and+Bowen+Zhou%2E+2025%2E+How+to+Synthesize+Text+Data+without+Model+Collapse%3F+Preprint%2C+arXi)
84. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, and 14 others. 2024. [BigCodeBench…](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/)
85. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-Tuning Language Models from Human Preferences. Preprint, arXiv:1909.08593. [find](https://scholar.google.com/scholar?q=Daniel+M%2E+Ziegler%2C+Nisan+Stiennon%2C+Jeffrey+Wu%2C+Tom+B%2E+Brown%2C+Alec+Radford%2C+Dario+Amodei%2C+Paul+Christiano%2C+and+Geoffrey+Irving%2E+2020%2E+Fine%2DTuning+Language+Models+from+Human+Preferences%2E+Preprint%2C+arXiv)

*Updated 31 October 2026*