---
title: "GitChameleon 2.0 — paper by Misra et al. (2026)"
description: "Existing benchmarks mostly test migration to newer or unseen versions, or score outputs by string… 70 references, cited by 1 paper here."
authors: ["Diganta Misra", "Nizar Islah", "Victor May", "Brice Rauby", "Zihan Wang", "Justine Gehring", "Antonio Orvieto", "Muawiz Sajjad Chaudhary", "Eilif B. Muller", "Irina Rish", "Samira Ebrahimi Kahou", "Massimo Caccia"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.2170/"
doi: "10.18653/v1/2026.acl-long.2170"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/misra-2026-gitchameleon-2-0-evaluating-ai/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/misra-2026-gitchameleon-2-0-evaluating-ai/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/) / GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities

# GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities

**Authors:** Diganta Misra, Nizar Islah, Victor May, Brice Rauby, [Zihan Wang](https://md.research.elamur.ai/free/en/authors/zihan-wang/), Justine Gehring, Antonio Orvieto, Muawiz Sajjad Chaudhary, Eilif B. Muller, Irina Rish, Samira Ebrahimi Kahou and Massimo Caccia · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.2170/) · [PDF](https://aclanthology.org/2026.acl-long.2170.pdf) · [DOI 10.18653/v1/2026.acl-long.2170](https://doi.org/10.18653/v1/2026.acl-long.2170) · [OpenAlex](https://openalex.org/W4414384564)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Code-generating LLMs are often asked to write code for a fixed, sometimes older, library release. Existing benchmarks mostly test migration to newer or unseen versions, or score outputs by string matching. GitChameleon 2.0 targets version-conditioned generation (VCG), where the model must write correct code for a specific version it has likely already seen in training. The challenge is choosing the right version, not data contamination.

The authors manually built 328 Python completion tasks around documented breaking changes in 26 libraries. Each task comes with hidden unit tests for scoring, a visible test for self-debugging, and documentation references for RAG. They evaluated LLMs under greedy decoding, Chain-of-Thought, Self-Debugging, RAG, multi-step agents and commercial coding assistants. Each run used both a version-specifying prompt and an execution environment pinned to that version.

Even strong enterprise models solve only about 48–51% of tasks at baseline. Self-debugging substantially reduces failures. The benchmark provides an execution-grounded way to measure whether code assistants respect dependency constraints.

## Key points
- The dataset has 328 problems across 26 libraries covering scientific computing, data science and web development. It expands the original GitChameleon, which had 116 problems, and took over 350 hours of manual curation.
- Library versions span 2014–2023, with most from 2021–2023. Argument or attribute changes are the most frequent change type, and semantic changes are the rarest.
- Performance is measured primarily by the share of problems whose solutions pass the hidden tests. API Hit Rate is reported as a secondary measure.
- GitChameleon 2.0 scores correlate moderately with SWE-Bench (ρ = 0.550, r = 0.675) and only weakly with LiveCodeBench (ρ = 0.214, r = 0.130).
- Self-Debugging cuts errors of every type by roughly 50–90%, including about 60–70% for AssertionError and TypeError. The stated limitations are Python-only coverage, no version-to-version translation tasks and no human baseline.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Université de Montréal](https://md.research.elamur.ai/free/en/universities/universite-de-montreal/) — University, Montreal, Canada · 68 works · 79,382 citations
- [McGill University](https://md.research.elamur.ai/free/en/universities/mcgill-university/) — University, Montreal, Canada · 87 works · 11,027 citations
- [Mila - Quebec Artificial Intelligence Institute](https://md.research.elamur.ai/free/en/labs/mila-quebec-artificial-intelligence-institute/) — Research institute, Montreal, Canada · 54 works · 13,565 citations
- [Polytechnique Montréal](https://md.research.elamur.ai/free/en/universities/polytechnique-montreal/) — University, Montreal, Canada · 9 works · 27,467 citations
- [ServiceNow](https://md.research.elamur.ai/free/en/companies/servicenow/) — Company, Santa Clara, United States · 10 works · 934 citations
- [ELLIS Institute Tübingen](https://md.research.elamur.ai/free/en/labs/ellis-institute-tubingen/) — Research institute, Tübingen, Germany · 2 works · 3 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Diganta Misra, Nizar Islah, Victor May, Brice Rauby, Zihan Wang, Justine Gehring, Antonio Orvieto, Muawiz Sajjad Chaudhary, Eilif B. Muller, Irina Rish, Samira Ebrahimi Kahou and Massimo Caccia. 2026. GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{misra2026gitchameleon,
  title = {{GitChameleon 2.0: Evaluating AI Code Generation Against Python Library Version Incompatibilities}},
  author = {Diganta Misra, Nizar Islah, Victor May, Brice Rauby, Zihan Wang, Justine Gehring, Antonio Orvieto, Muawiz Sajjad Chaudhary, Eilif B. Muller, Irina Rish, Samira Ebrahimi Kahou and Massimo Caccia},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2170},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2170}
}
```

- [← previous Generating Effective CoT Traces for Mitigating Causal Hallucination](https://md.research.elamur.ai/free/en/papers/zhao-2026-generating-effective-cot-traces-mitigating/)
- [next → Glite at BEA 2026 Shared Task 1: Holistic Difficulty Models Dominate, Feature Engineering Closes the Gap in L1-Aware Vocabulary Difficulty Prediction](https://md.research.elamur.ai/free/en/papers/philippov-2026-glite-bea-2026-shared-task/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/7/)

## References

70 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Google Cloud. 2025. Gemini 2.5 on Vertex AI: Pro, Flash & Model Optimizer Live. [https://cloud.google.com/blog/products/ai-machine-lea](https://cloud.google.com/blog/products/ai-machine-lea) rning/gemini-2-5-pro-flash-on-vertex-ai. Discusses Gemini 2.5 Pro and Gemini 2.5 Flash. [↗](https://cloud.google.com/blog/products/ai-machine-lea)
2. Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. 2019. Gradio: Hassle-free sharing and testing of ML models in the wild. Preprint, arXiv:1906.02569. — [arXiv:1906.02569](https://arxiv.org/abs/1906.02569) [find](https://scholar.google.com/scholar?q=Abubakar+Abid%2C+Ali+Abdalla%2C+Ali+Abid%2C+Dawood+Khan%2C+Abdulrahman+Alfozan%2C+and+James+Zou%2E+2019%2E+Gradio%3A+Hassle%2Dfree+sharing+and+testing+of+ML+models+in+the+wild%2E+Preprint%2C+arXiv%3A1906%2E02569%2E+%E2%80%94+%5BarXiv%3A1906)
3. Meta AI. 2025. Everything we announced at our first-ever LlamaCon. [https://ai.meta.com/blog/lla](https://ai.meta.com/blog/lla) macon-llama-news/. Discusses Llama 3.3 Instruct Turbo and Llama 4 Maverick. [↗](https://ai.meta.com/blog/lla)
4. Mohannad Alhanahnah, Yazan Boshmaf, and Benoit Baudry. 2024. DepsRAG: Towards managing software dependencies using large language models. arXiv preprint arXiv:2405.20455v2. — [arXiv:2405.20455](https://arxiv.org/abs/2405.20455) [find](https://scholar.google.com/scholar?q=Mohannad+Alhanahnah%2C+Yazan+Boshmaf%2C+and+Benoit+Baudry%2E+2024%2E+DepsRAG%3A+Towards+managing+software+dependencies+using+large+language+models%2E+arXiv+preprint+arXiv%3A2405%2E20455v2%2E+%E2%80%94+%5BarXiv%3A2405%2E20455%5D%28https%3A)
5. Matteo Ciniselli, Alberto Martin-Lopez, and Gabriele Bavota. 2024. On the generalizability of deep learning-based code completion across programming language versions. Preprint, arXiv:2403.15149. — [arXiv:2403.15149](https://arxiv.org/abs/2403.15149) [find](https://scholar.google.com/scholar?q=Matteo+Ciniselli%2C+Alberto+Martin%2DLopez%2C+and+Gabriele+Bavota%2E+2024%2E+On+the+generalizability+of+deep+learning%2Dbased+code+completion+across+programming+language+versions%2E+Preprint%2C+arXiv%3A2403%2E15149%2E+%E2%80%94+%5Ba)
6. Team Cohere, :, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, and 211 others. 2025.… — [arXiv:2504.00698](https://arxiv.org/abs/2504.00698) [find](https://scholar.google.com/scholar?q=Team+Cohere%2C+%3A%2C+Aakanksha%2C+Arash+Ahmadian%2C+Marwan+Ahmed%2C+Jay+Alammar%2C+Milad+Alizadeh%2C+Yazeed+Alnumay%2C+Sophia+Althammer%2C+Arkady+Arkhangorodsky%2C+Viraat+Aryabumi%2C+Dennis+Aumiller%2C+Rapha%C3%ABl+Avalos%2C+Zahara+)
7. Anthropic. 2025. Claude 3.7 Sonnet and Claude Code. [https://www.anthropic.com/news/claude-3-7](https://www.anthropic.com/news/claude-3-7) -sonnet. [↗](https://www.anthropic.com/news/claude-3-7)
8. Forbes Technology Council. 2024. Revolutionizing software development with large language models. [https://www.forbes.com/councils/forbeste](https://www.forbes.com/councils/forbeste) chcouncil/2024/03/20/revolutionizing-sof tware-development-with-large-language-mod els/. [↗](https://www.forbes.com/councils/forbeste)
9. Arcee. Model Selection \| Arcee AI Documentation — docs.arcee.ai. [https://docs.arcee.ai/arcee-c](https://docs.arcee.ai/arcee-c) onductor/arcee-small-language-models/mode l-selection\#caller-large-tool-use-and-fun ction-call. [Accessed 15-07-2025]. [↗](https://docs.arcee.ai/arcee-c)
10. Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems (NeurIPS). [find](https://scholar.google.com/scholar?q=Tri+Dao%2C+Daniel+Y%2E+Fu%2C+Stefano+Ermon%2C+Atri+Rudra%2C+and+Christopher+R%C3%A9%2E+2022%2E+FlashAttention%3A+Fast+and+memory%2Defficient+exact+attention+with+IO%2Dawareness%2E+In+Advances+in+Neural+Information+Processing+Sy)
11. Farnaz Behrang, Zhizhou Zhang, Georgian-Vlad Saioc, Peng Liu, and Milind Chabbi. 2025. Dr.fix: Automatically fixing data races at industry scale. Preprint, arXiv:2504.15637. — [arXiv:2504.15637](https://arxiv.org/abs/2504.15637) [find](https://scholar.google.com/scholar?q=Farnaz+Behrang%2C+Zhizhou+Zhang%2C+Georgian%2DVlad+Saioc%2C+Peng+Liu%2C+and+Milind+Chabbi%2E+2025%2E+Dr%2Efix%3A+Automatically+fixing+data+races+at+industry+scale%2E+Preprint%2C+arXiv%3A2504%2E15637%2E+%E2%80%94+%5BarXiv%3A2504%2E15637%5D%28https)
12. Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Andreas Joly, Bertrand Druillette, Gael Varoquaux, and Marion Gramfort. 2013. API design for machine… — [arXiv:1309.0238](https://arxiv.org/abs/1309.0238) [find](https://scholar.google.com/scholar?q=Lars+Buitinck%2C+Gilles+Louppe%2C+Mathieu+Blondel%2C+Fabian+Pedregosa%2C+Andreas+Mueller%2C+Olivier+Grisel%2C+Vlad+Niculae%2C+Peter+Prettenhofer%2C+Alexandre+Gramfort%2C+Jaques+Grobler%2C+Robert+Layton%2C+Jake+VanderPlas%2C+)
13. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, and 39 others. 202… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
14. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching large language models to self-debug. Preprint, arXiv:2304.05128. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Teaching+large+language+models+to+self%2Ddebug%2E+Preprint%2C+arXiv%3A2304%2E05128%2E)
15. DeepSeek-AI. 2025. [Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2025-deepseek-r1-incentivizing-reasoning-capability/). Preprint, arXiv:2501.12948.
16. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… [find](https://scholar.google.com/scholar?q=DeepSeek%2DAI%2C+Aixin+Liu%2C+Bei+Feng%2C+Bing+Xue%2C+Bingxuan+Wang%2C+Bochao+Wu%2C+Chengda+Lu%2C+Chenggang+Zhao%2C+Chengqi+Deng%2C+Chenyu+Zhang%2C+Chong+Ruan%2C+Damai+Dai%2C+Daya+Guo%2C+Dejian+Yang%2C+Deli+Chen%2C+Dongjie+Ji%2C+Erhan)
17. DuckDuckGo. 2025. DuckDuckGo: Privacy, simplified. [https://duckduckgo.com/](https://duckduckgo.com/). [↗](https://duckduckgo.com/)
18. Lishui Fan, Mouxiang Chen, and Zhongxin Liu. 2024. Self-explained keywords empower large language models for code generation. Preprint, arXiv:2410.15966. — [arXiv:2410.15966](https://arxiv.org/abs/2410.15966) [find](https://scholar.google.com/scholar?q=Lishui+Fan%2C+Mouxiang+Chen%2C+and+Zhongxin+Liu%2E+2024%2E+Self%2Dexplained+keywords+empower+large+language+models+for+code+generation%2E+Preprint%2C+arXiv%3A2410%2E15966%2E+%E2%80%94+%5BarXiv%3A2410%2E15966%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F241)
19. Google. 2025. Grounding with Google Search \| Gemini API. [https://ai.google.dev/gemini-api/doc](https://ai.google.dev/gemini-api/doc) s/grounding. [↗](https://ai.google.dev/gemini-api/doc)
20. Aric A Hagberg, Daniel A Schult, and Pieter J Swart. 2008. Exploring network structure, dynamics, and function using NetworkX. In Proceedings of the 7th Python in Science Conference, pages 11–15. [find](https://scholar.google.com/scholar?q=Aric+A+Hagberg%2C+Daniel+A+Schult%2C+and+Pieter+J+Swart%2E+2008%2E+Exploring+network+structure%2C+dynamics%2C+and+function+using+NetworkX%2E+In+Proceedings+of+the+7th+Python+in+Science+Conference%2C+pages+11%E2%80%9315%2E)
21. Charles R Harris, K Jarrod Millman, Stéfan J van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J Smith, Robert Kern, Matti Picus, Changqing Hoyer, Marten H van Kerkwijk, Alex Brett, Andrew Wen, Pete Zhang, Joe Igoe, Keith Featherstone… — [Array Programming with NumPy](https://md.research.elamur.ai/free/en/papers/harris-2020-array-programming-numpy/)
22. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). NeurIPS.
23. J. D. Hunter. 2007. Matplotlib: A 2D graphics environment. Computing in Science & Engineering, 9(3):90–95. [find](https://scholar.google.com/scholar?q=J%2E+D%2E+Hunter%2E+2007%2E+Matplotlib%3A+A+2D+graphics+environment%2E+Computing+in+Science+%26+Engineering%2C+9%283%29%3A90%E2%80%9395%2E)
24. Amazon Artificial General Intelligence. 2024. The amazon nova family of models: Technical report and model card. Amazon Technical Reports. [find](https://scholar.google.com/scholar?q=Amazon+Artificial+General+Intelligence%2E+2024%2E+The+amazon+nova+family+of+models%3A+Technical+report+and+model+card%2E+Amazon+Technical+Reports%2E)
25. Mohayeminul Islam, Ajay Kumar Jha, Sarah Nadi, and Ildar Akhmetov. 2023. Pymigbench: A benchmark for python library migration. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), pages 511–515. [find](https://scholar.google.com/scholar?q=Mohayeminul+Islam%2C+Ajay+Kumar+Jha%2C+Sarah+Nadi%2C+and+Ildar+Akhmetov%2E+2023%2E+Pymigbench%3A+A+benchmark+for+python+library+migration%2E+In+2023+IEEE%2FACM+20th+International+Conference+on+Mining+Software+Reposit)
26. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. [Livecodebench: Holistic and contamination free evaluation of large language models for code](https://md.research.elamur.ai/free/en/papers/jain-2024-livecodebench-holistic-contamination-free-evaluation/). arXiv preprint arXiv:2403.07974.
27. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. [SWE-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations.
28. Haolin Jin, Zechao Sun, and Huaming Chen. 2024. Rgd: Multi-llm based agent debugger via refinement and generation guidance. Preprint, arXiv:2410.01242. — [arXiv:2410.01242](https://arxiv.org/abs/2410.01242) [find](https://scholar.google.com/scholar?q=Haolin+Jin%2C+Zechao+Sun%2C+and+Huaming+Chen%2E+2024%2E+Rgd%3A+Multi%2Dllm+based+agent+debugger+via+refinement+and+generation+guidance%2E+Preprint%2C+arXiv%3A2410%2E01242%2E+%E2%80%94+%5BarXiv%3A2410%2E01242%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2410%2E)
29. Kelsey Jordahl, Joris Van den Bossche, Martin Fleis-chmann, Jacob Wasserman, James McBride, Jeffrey Gerard, Jeff Tratner, Matthew Perry, Adrian Garcia Badaracco, Carson Farmer, Geir Arne Hjelle, Alan D. Snow, Micah Cochran, Sean Gillies, Lucas Culbertson, Matt Bartos, Nick Eubank, maxalbert, Aleksey… [find](https://scholar.google.com/scholar?q=Kelsey+Jordahl%2C+Joris+Van+den+Bossche%2C+Martin+Fleis%2Dchmann%2C+Jacob+Wasserman%2C+James+McBride%2C+Jeffrey+Gerard%2C+Jeff+Tratner%2C+Matthew+Perry%2C+Adrian+Garcia+Badaracco%2C+Carson+Farmer%2C+Geir+Arne+Hjelle%2C+Alan+)
30. Kat Kampf. 2025. Create and edit images with Gemini 2.0 in preview. [https://developers.googleblo](https://developers.googleblo) g.com/en/generate-images-gemini-2-0-flash -preview/. Discusses Gemini 2.0 Flash. [↗](https://developers.googleblo)
31. Paul Kassianik, Baturay Saglam, Alexander Chen, Blaine Nelson, Anu Vellore, Massimo Aufiero, Fraser Burch, Dhruv Kedia, Avi Zohary, Sajana Weerawardhena, Aman Priyanshu, Adam Swanda, Amy Chang, Hyrum Anderson, Kojin Oshiba, Omar Santos, Yaron Singer, and Amin Karbasi. 2025. Llama-3.1-FoundationAI-Se… — [arXiv:2504.21039](https://arxiv.org/abs/2504.21039) [find](https://scholar.google.com/scholar?q=Paul+Kassianik%2C+Baturay+Saglam%2C+Alexander+Chen%2C+Blaine+Nelson%2C+Anu+Vellore%2C+Massimo+Aufiero%2C+Fraser+Burch%2C+Dhruv+Kedia%2C+Avi+Zohary%2C+Sajana+Weerawardhena%2C+Aman+Priyanshu%2C+Adam+Swanda%2C+Amy+Chang%2C+Hyrum+)
32. Sachit Kuhar, Wasi Uddin Ahmad, Zijian Wang, Nihal Jain, Haifeng Qian, Baishakhi Ray, Murali Krishna Ramanathan, Xiaofei Ma, and Anoop Deoras. 2024. Libevolutioneval: A benchmark and study for version-specific code generation. Preprint, arXiv:2412.04478. [find](https://scholar.google.com/scholar?q=Sachit+Kuhar%2C+Wasi+Uddin+Ahmad%2C+Zijian+Wang%2C+Nihal+Jain%2C+Haifeng+Qian%2C+Baishakhi+Ray%2C+Murali+Krishna+Ramanathan%2C+Xiaofei+Ma%2C+and+Anoop+Deoras%2E+2024%2E+Libevolutioneval%3A+A+benchmark+and+study+for+version)
33. Stefano Lambiase, Gemma Catolino, Fabio Palomba, Filomena Ferrucci, and Daniel Russo. 2025. Exploring individual factors in the adoption of llms for specific software engineering tasks. Preprint, arXiv:2504.02553. — [arXiv:2504.02553](https://arxiv.org/abs/2504.02553) [find](https://scholar.google.com/scholar?q=Stefano+Lambiase%2C+Gemma+Catolino%2C+Fabio+Palomba%2C+Filomena+Ferrucci%2C+and+Daniel+Russo%2E+2025%2E+Exploring+individual+factors+in+the+adoption+of+llms+for+specific+software+engineering+tasks%2E+Preprint%2C+arXi)
34. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. [Retrieval-augmented Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Proc…
35. James Li. 2024. ReAct vs Plan-and-Execute: A Practical Comparison of LLM Agent Patterns. [https://dev.to/jamesli](https://dev.to/jamesli). [↗](https://dev.to/jamesli)
36. Linxi Liang, Jing Gong, Mingwei Liu, Chong Wang, Guangsheng Ou, Yanlin Wang, Xin Peng, and Zibin Zheng. 2025. Rustevo: An evolving benchmark for api evolution in llm-based rust code generation. Preprint, arXiv:2503.16922. — [arXiv:2503.16922](https://arxiv.org/abs/2503.16922) [find](https://scholar.google.com/scholar?q=Linxi+Liang%2C+Jing+Gong%2C+Mingwei+Liu%2C+Chong+Wang%2C+Guangsheng+Ou%2C+Yanlin+Wang%2C+Xin+Peng%2C+and+Zibin+Zheng%2E+2025%2E+Rustevo%3A+An+evolving+benchmark+for+api+evolution+in+llm%2Dbased+rust+code+generation%2E+Prepri)
37. Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, and 3 others. 2024. [Jamba:…](https://md.research.elamur.ai/free/en/papers/lieber-2024-jamba-hybrid-transformer-mamba-language/)
38. Yue Liu, Chakkrit Tantithamthavorn, Yonghui Liu, Patanamon Thongtanunam, and Li Li. 2024. Automatically recommend code updates: Are we there yet? Preprint, arXiv:2209.07048. — [arXiv:2209.07048](https://arxiv.org/abs/2209.07048) [find](https://scholar.google.com/scholar?q=Yue+Liu%2C+Chakkrit+Tantithamthavorn%2C+Yonghui+Liu%2C+Patanamon+Thongtanunam%2C+and+Li+Li%2E+2024%2E+Automatically+recommend+code+updates%3A+Are+we+there+yet%3F+Preprint%2C+arXiv%3A2209%2E07048%2E+%E2%80%94+%5BarXiv%3A2209%2E07048%5D%28https)
39. Zeyu Leo Liu, Shrey Pandit, Xi Ye, Eunsol Choi, and Greg Durrett. 2025. Codeupdatearena: Benchmarking knowledge editing on API updates. [find](https://scholar.google.com/scholar?q=Zeyu+Leo+Liu%2C+Shrey+Pandit%2C+Xi+Ye%2C+Eunsol+Choi%2C+and+Greg+Durrett%2E+2025%2E+Codeupdatearena%3A+Benchmarking+knowledge+editing+on+API+updates%2E)
40. Edward Loper and Steven Bird. 2002. NLTK: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, pages 63–70, Philadelphia, Pennsylvania, USA. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Edward+Loper+and+Steven+Bird%2E+2002%2E+NLTK%3A+The+natural+language+toolkit%2E+In+Proceedings+of+the+ACL%2D02+Workshop+on+Effective+Tools+and+Methodologies+for+Teaching+Natural+Language+Processing+and+Computat)
41. Stephan Lukasczyk and Gordon Fraser. 2022. Pynguin: Automated unit test generation for python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 168–172. [find](https://scholar.google.com/scholar?q=Stephan+Lukasczyk+and+Gordon+Fraser%2E+2022%2E+Pynguin%3A+Automated+unit+test+generation+for+python%2E+In+Proceedings+of+the+ACM%2FIEEE+44th+International+Conference+on+Software+Engineering%3A+Companion+Proceedin)
42. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. WizardCoder: Empowering code large language models with Evol-Instruct. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2023%2E+WizardCoder%3A+Empowering+code+large+language+models+with+Evol%2DInstruct%2E)
43. Wes McKinney. 2010. Data Structures for Statistical Computing in Python. In Proceedings of the 9th Python in Science Conference, pages 51–56. [find](https://scholar.google.com/scholar?q=Wes+McKinney%2E+2010%2E+Data+Structures+for+Statistical+Computing+in+Python%2E+In+Proceedings+of+the+9th+Python+in+Science+Conference%2C+pages+51%E2%80%9356%2E)
44. Mistral AI. 2025. Medium is the new large: Introducing mistral medium 3. [https://mistral.ai/news/mi](https://mistral.ai/news/mi) stral-medium-3. Accessed: 2025-05-17. [↗](https://mistral.ai/news/mi)
45. OpenAI. 2024. [GPT-4o System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-gpt-4o-system-card/). arXiv preprint arXiv:2410.21276. Cited for GPT-4o.
46. OpenAI. 2024. New embedding models and api updates. [https://openai.com/index/new-embedding-m](https://openai.com/index/new-embedding-m) odels-and-api-updates/. Accessed: 2025-07-28. [↗](https://openai.com/index/new-embedding-m)
47. OpenAI. 2024. [OpenAI o1 System Card](https://md.research.elamur.ai/free/en/papers/openai-2024-openai-o1-system-card/). [https://openai.com/index/openai-o1-system-card/](https://openai.com/index/openai-o1-system-card/). Discusses the o1 model series, including o1 and mentioning o3-mini. [↗](https://openai.com/index/openai-o1-system-card/)
48. OpenAI. 2025a. Introducing GPT-4.1 in the API. [https://openai.com/index/gpt-4-1/](https://openai.com/index/gpt-4-1/). Discusses GPT-4.1, GPT-4.1 mini, and GPT-4.1 nano. [↗](https://openai.com/index/gpt-4-1/)
49. OpenAI. 2025b. Introducing GPT-4.5. https://open ai.com/index/introducing-gpt-4-5/. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2025b%2E+Introducing+GPT%2D4%2E5%2E+https%3A%2F%2Fopen+ai%2Ecom%2Findex%2Fintroducing%2Dgpt%2D4%2D5%2F%2E)
50. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, and 2 others. 2019.… — [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
51. Perplexity AI. 2024. Getting started with Perplexity. [https://www.perplexity.ai/hub/blog/getti](https://www.perplexity.ai/hub/blog/getti) ng-started-with-perplexity. [↗](https://www.perplexity.ai/hub/blog/getti)
52. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. [find](https://scholar.google.com/scholar?q=Qwen%2C+%3A%2C+An+Yang%2C+Baosong+Yang%2C+Beichen+Zhang%2C+Binyuan+Hui%2C+Bo+Zheng%2C+Bowen+Yu%2C+Chengyuan+Li%2C+Dayiheng+Liu%2C+Fei+Huang%2C+Haoran+Wei%2C+Huan+Lin%2C+Jian+Yang%2C+Jianhong+Tu%2C+Jianwei+Zhang%2C+Jianxin+Yang%2C+Jiaxi+)
53. Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. 2025. ‘smolagents‘: a smol library to build great agentic systems. [https://github.com/huggingfa](https://github.com/huggingfa) ce/smolagents. [↗](https://github.com/huggingfa)
54. Aditi Singh, Abul Ehtesham, Saket Kumar, and Tala Talaei Khoei. 2025. Agentic retrieval-augmented generation: A survey on agentic rag. Preprint, arXiv:2501.09136. — [arXiv:2501.09136](https://arxiv.org/abs/2501.09136) [find](https://scholar.google.com/scholar?q=Aditi+Singh%2C+Abul+Ehtesham%2C+Saket+Kumar%2C+and+Tala+Talaei+Khoei%2E+2025%2E+Agentic+retrieval%2Daugmented+generation%3A+A+survey+on+agentic+rag%2E+Preprint%2C+arXiv%3A2501%2E09136%2E+%E2%80%94+%5BarXiv%3A2501%2E09136%5D%28https%3A%2F%2Farxiv%2Eor)
55. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. 2023. [Roformer: Enhanced transformer with rotary position embedding](https://md.research.elamur.ai/free/en/papers/su-2021-roformer-enhanced-transformer-rotary-position/). Preprint, arXiv:2104.09864.
56. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick, Lexi Walker, Cosmin Paduraru, Christina Sorokin, and 1118 others. 2024. G… — [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/)
57. The pandas development team. 2020. [dev/pandas: Pandas](https://md.research.elamur.ai/free/en/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/). pandas-Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, and 1 others. 2020. Scipy 1.0: fundamental algorithms for scientific compu…
58. Chaozheng Wang, Shuzheng Gao, Cuiyun Gao, Wenxuan Wang, Chun Yong Chong, Shan Gao, and Michael R. Lyu. 2024a. A systematic evaluation of large code models in api suggestion: When, which, and how. Preprint, arXiv:2409.13178. — [arXiv:2409.13178](https://arxiv.org/abs/2409.13178) [find](https://scholar.google.com/scholar?q=Chaozheng+Wang%2C+Shuzheng+Gao%2C+Cuiyun+Gao%2C+Wenxuan+Wang%2C+Chun+Yong+Chong%2C+Shan+Gao%2C+and+Michael+R%2E+Lyu%2E+2024a%2E+A+systematic+evaluation+of+large+code+models+in+api+suggestion%3A+When%2C+which%2C+and+how%2E+Prep)
59. Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2024b. How and Why LLMs Use Deprecated APIs in Code Completion? an Empirical Study. arXiv preprint arXiv:2312.14617. — [arXiv:2312.14617](https://arxiv.org/abs/2312.14617) [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+Kaifeng+Huang%2C+Jian+Zhang%2C+Yebo+Feng%2C+Lyuye+Zhang%2C+Yang+Liu%2C+and+Xin+Peng%2E+2024b%2E+How+and+Why+LLMs+Use+Deprecated+APIs+in+Code+Completion%3F+an+Empirical+Study%2E+arXiv+preprint+arXiv%3A2312%2E146)
60. Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2025a. LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion . In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 781–781, Los Alamitos, CA, US… [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+Kaifeng+Huang%2C+Jian+Zhang%2C+Yebo+Feng%2C+Lyuye+Zhang%2C+Yang+Liu%2C+and+Xin+Peng%2E+2025a%2E+LLMs+Meet+Library+Evolution%3A+Evaluating+Deprecated+API+Usage+in+LLM%2Dbased+Code+Completion+%2E+In+2025+IEEE%2FA)
61. Rafiqul Rabin, Jesse Hostetler, Sean McGregor, Brett Weir, and Nick Judd. 2025. Sandboxeval: Towards securing test environment for untrusted code. Preprint, arXiv:2504.00018. — [arXiv:2504.00018](https://arxiv.org/abs/2504.00018) [find](https://scholar.google.com/scholar?q=Rafiqul+Rabin%2C+Jesse+Hostetler%2C+Sean+McGregor%2C+Brett+Weir%2C+and+Nick+Judd%2E+2025%2E+Sandboxeval%3A+Towards+securing+test+environment+for+untrusted+code%2E+Preprint%2C+arXiv%3A2504%2E00018%2E+%E2%80%94+%5BarXiv%3A2504%2E00018%5D%28http)
62. Chong Wang, Kaifeng Huang, Jian Zhang, Yebo Feng, Lyuye Zhang, Yang Liu, and Xin Peng. 2025b. Llms meet library evolution: Evaluating deprecated api usage in llm-based code completion. Preprint, arXiv:2406.09834. — [arXiv:2406.09834](https://arxiv.org/abs/2406.09834) [find](https://scholar.google.com/scholar?q=Chong+Wang%2C+Kaifeng+Huang%2C+Jian+Zhang%2C+Yebo+Feng%2C+Lyuye+Zhang%2C+Yang+Liu%2C+and+Xin+Peng%2E+2025b%2E+Llms+meet+library+evolution%3A+Evaluating+deprecated+api+usage+in+llm%2Dbased+code+completion%2E+Preprint%2C+arXiv)
63. Reka. RekaAI/reka-flash-3 · Hugging Face — huggingface.co. [https://huggingface.co/RekaAI/reka](https://huggingface.co/RekaAI/reka) -flash-3. [Accessed 15-07-2025]. [↗](https://huggingface.co/RekaAI/reka)
64. Xingyao Wang. 2025. Introducing openhands lm 32b – a strong, open coding agent model. All Hands AI Blog. [find](https://scholar.google.com/scholar?q=Xingyao+Wang%2E+2025%2E+Introducing+openhands+lm+32b+%E2%80%93+a+strong%2C+open+coding+agent+model%2E+All+Hands+AI+Blog%2E)
65. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. [Chain-of-thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). Preprint, arXiv:2201.11903.
66. Tongtong Wu, Weigang Wu, Xingyu Wang, Kang Xu, Suyu Ma, Bo Jiang, Ping Yang, Zhenchang Xing, Yuan-Fang Li, and Gholamreza Haffari. 2024. Versi-Code: Towards version-controllable code generation. xAI. 2025. Grok-3. Official xAI announcement. Accessed May 17, 2025. [find](https://scholar.google.com/scholar?q=Tongtong+Wu%2C+Weigang+Wu%2C+Xingyu+Wang%2C+Kang+Xu%2C+Suyu+Ma%2C+Bo+Jiang%2C+Ping+Yang%2C+Zhenchang+Xing%2C+Yuan%2DFang+Li%2C+and+Gholamreza+Haffari%2E+2024%2E+Versi%2DCode%3A+Towards+version%2Dcontrollable+code+generation%2E+xAI%2E+)
67. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. [Qwen3 technical report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). Preprint, arXiv:2505.09388.
68. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. [React: Synergizing reasoning and acting in language models](https://md.research.elamur.ai/free/en/papers/yao-2022-react-synergizing-reasoning-acting-language/). In The Eleventh International Conference on Learning Representations.
69. Sixiang Ye, Zeyu Sun, Guoqing Wang, Liwei Guo, Qingyuan Liang, Zheng Li, and Yong Liu. 2025. Prompt alchemy: Automatic prompt refinement for enhancing code generation. Preprint, arXiv:2503.11085. — [arXiv:2503.11085](https://arxiv.org/abs/2503.11085) [find](https://scholar.google.com/scholar?q=Sixiang+Ye%2C+Zeyu+Sun%2C+Guoqing+Wang%2C+Liwei+Guo%2C+Qingyuan+Liang%2C+Zheng+Li%2C+and+Yong+Liu%2E+2025%2E+Prompt+alchemy%3A+Automatic+prompt+refinement+for+enhancing+code+generation%2E+Preprint%2C+arXiv%3A2503%2E11085%2E+%E2%80%94+%5Ba)
70. Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2022. DocPrompting: Generating code by retrieving the docs. [find](https://scholar.google.com/scholar?q=Shuyan+Zhou%2C+Uri+Alon%2C+Frank+F+Xu%2C+Zhiruo+Wang%2C+Zhengbao+Jiang%2C+and+Graham+Neubig%2E+2022%2E+DocPrompting%3A+Generating+code+by+retrieving+the+docs%2E)

*Updated 1 November 2026*