---
title: "SpecAgent — paper by Ma et al. (2026) — summary and ideas"
description: "To avoid this, they build a leakage-free synthetic benchmark from REPOCOD. 44 references, cited by 1 paper here."
authors: ["George Ma", "Anurag Koul", "Qi Chen", "Yawen Wu", "Sachit Kuhar", "Yu Yu", "Aritra Sengupta", "Varun Kumar", "Murali Krishna Ramanathan"]
year: 2026
venue: "Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2026.acl-long.786/"
doi: "10.18653/v1/2026.acl-long.786"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2026-specagent-speculative-retrieval-forecasting-agent/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2026-specagent-speculative-retrieval-forecasting-agent/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/) / SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion

# SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion

**Authors:** George Ma, Anurag Koul, [Qi Chen](https://md.research.elamur.ai/free/en/authors/qi-chen/), Yawen Wu, Sachit Kuhar, Yu Yu, Aritra Sengupta, [Varun Kumar](https://md.research.elamur.ai/free/en/authors/varun-kumar/) and [Murali Krishna Ramanathan](https://md.research.elamur.ai/free/en/authors/murali-krishna-ramanathan/) · **Year:** 2026 · **Venue:** [Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2026](https://md.research.elamur.ai/free/en/venues/acl/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.acl-long.786/) · [PDF](https://aclanthology.org/2026.acl-long.786.pdf) · [DOI 10.18653/v1/2026.acl-long.786](https://doi.org/10.18653/v1/2026.acl-long.786) · [OpenAlex](https://openalex.org/W4416054658)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · 1 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Repository-level code completion depends on project-specific knowledge such as private APIs and cross-file dependencies. Fetching that knowledge at inference time (BM25, dense embeddings) adds delay that hurts interactive in-IDE inline completion. In the authors' tests this delay reached 9 seconds. SpecAgent moves the work to repository indexing. An LLM agent with file-reading and bash tools explores the codebase in the background and prepares context blocks for every file. Some blocks are retrieved cross-file snippets. Others are speculative predictions of functions the developer is likely to write next. At completion time the code model simply receives these blocks, so no lookup is needed.

The authors also show that existing benchmarks suffer from future-context leakage: callers and tests elsewhere in the repository reveal the target function's signature and usage, which inflates scores. To avoid this, they build a leakage-free synthetic benchmark from REPOCOD. With Qwen3-8B and Qwen3-30B-A3B as completion models, SpecAgent beats the strongest baselines by 9–11 points pass@1 and adds no retrieval latency at inference.

## Key points
- **Main gains:** Qwen3-8B improves by about 10–11 points absolute (58% relative) and Qwen3-30B-A3B by about 9–10 points (48% relative). The baselines are BM25, dense retrieval (UniXcoder, CodeSage) and RepoMap. The cost is a one-time background indexing step of roughly 50 seconds. The agent runs on Claude 3.7 Sonnet.
- **Benchmark:** REPOCOD was modified with a function-removal agent to simulate repository states at indexing time. An Oracle Agent with access to the ground truth provides an upper bound.
- **Evidence of leakage:** Running SpecAgent in the leaky inference-time setting raises pass@1 from 27.86 to 30.92 (Qwen3-8B) and from 27.96 to 34.29 (Qwen3-30B-A3B). The authors consider these higher numbers unrealistic.
- **Ablations:**
  - Both components are needed, and the Forecaster Agent alone beats the Retriever Agent alone.
  - Three prediction blocks work best (27.86 pass@1). With zero prediction blocks the score is 20.92, and with twelve it is 22.35.
  - Adding BM25 or UniXcoder contexts to SpecAgent's blocks lowers pass@1 to 25.10 and 25.31.
- **Setup and limits:**
  - A maximum of 10K tokens applies separately to the left context, the right context and the cross-file context, and experiments ran on 8 A100 GPUs.
  - The benchmark is synthetic rather than drawn from real repository histories.
  - Indexing overhead may matter for very large or fast-changing repositories.
  - Only two models were tested.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 175 works · 227,440 citations
- [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) — Company, Seattle, United States · 112 works · 40,565 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🔍 Information retrieval & search](https://md.research.elamur.ai/free/en/topics/information-retrieval/) · 473 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: George Ma, Anurag Koul, Qi Chen, Yawen Wu, Sachit Kuhar, Yu Yu, Aritra Sengupta, Varun Kumar and Murali Krishna Ramanathan. 2026. SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ma2026specagent,
  title = {{SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion}},
  author = {George Ma, Anurag Koul, Qi Chen, Yawen Wu, Sachit Kuhar, Yu Yu, Aritra Sengupta, Varun Kumar and Murali Krishna Ramanathan},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.786},
  url = {https://doi.org/10.18653/v1/2026.acl-long.786}
}
```

- [← previous Speak in Context: Multilingual ASR with Speech–Context Alignment via Contrastive Learning](https://md.research.elamur.ai/free/en/papers/zhang-2026-speak-context-multilingual-asr-speech/)
- [next → Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation](https://md.research.elamur.ai/free/en/papers/zhang-2026-specializing-large-models-oracle-bone/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/17/)

## References

44 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547. [find](https://scholar.google.com/scholar?q=Natural+Languages%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2020%2C+pages+1536%E2%80%931547%2E)
2. Aider. 2023. Repository map. Blog post. [find](https://scholar.google.com/scholar?q=Aider%2E+2023%2E+Repository+map%2E+Blog+post%2E)
3. Anthropic. 2025. Claude 3.7 Sonnet and Claude Code. Blog post. [find](https://scholar.google.com/scholar?q=Anthropic%2E+2025%2E+Claude+3%2E7+Sonnet+and+Claude+Code%2E+Blog+post%2E)
4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. [Program Synthesis with Large Language Models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
5. Max Brunsfeld, Patrick Thomson, Andrew Hlynskyi, Josh Vera, Phil Turnbull, Timothy Clem, Douglas Creager, Andrew Helwer, Rob Rix, Hendrik van Antwerpen, Michael Davis, Ika, Tuan-Anh Nguyen, Stafford Brunk, Niranjan Hasabnis, bfredl, Mingkai Dong, Vladimir Panteleev, ikrima, and 10 others. 2019. Tree… [find](https://scholar.google.com/scholar?q=Max+Brunsfeld%2C+Patrick+Thomson%2C+Andrew+Hlynskyi%2C+Josh+Vera%2C+Phil+Turnbull%2C+Timothy+Clem%2C+Douglas+Creager%2C+Andrew+Helwer%2C+Rob+Rix%2C+Hendrik+van+Antwerpen%2C+Michael+Davis%2C+Ika%2C+Tuan%2DAnh+Nguyen%2C+Stafford+B)
6. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
7. Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2023. Teaching Large Language Models to Self-Debug. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Xinyun+Chen%2C+Maxwell+Lin%2C+Nathanael+Sch%C3%A4rli%2C+and+Denny+Zhou%2E+2023%2E+Teaching+Large+Language+Models+to+Self%2DDebug%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
8. Ken Deng, Jiaheng Liu, He Zhu, Congnan Liu, Jingxin Li, Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin, and 1 others. 2024. R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models. In CoRR. [find](https://scholar.google.com/scholar?q=Ken+Deng%2C+Jiaheng+Liu%2C+He+Zhu%2C+Congnan+Liu%2C+Jingxin+Li%2C+Jiakai+Wang%2C+Peng+Zhao%2C+Chenchen+Zhang%2C+Yanan+Wu%2C+Xueqiao+Yin%2C+and+1+others%2E+2024%2E+R2C2%2DCoder%3A+Enhancing+and+Benchmarking+Real%2Dworld+Repository%2D)
9. Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang. 2023. Large Language Models Are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 423–43… [find](https://scholar.google.com/scholar?q=Yinlin+Deng%2C+Chunqiu+Steven+Xia%2C+Haoran+Peng%2C+Chenyuan+Yang%2C+and+Lingming+Zhang%2E+2023%2E+Large+Language+Models+Are+Zero%2DShot+Fuzzers%3A+Fuzzing+Deep%2DLearning+Libraries+via+Large+Language+Models%2E+In+Procee)
10. Akash Dhruv and Anshu Dubey. 2025. Leveraging large language models for code translation and software development in scientific computing. In Proceedings of the Platform for Advanced Scientific Computing Conference, pages 1–9. [find](https://scholar.google.com/scholar?q=Akash+Dhruv+and+Anshu+Dubey%2E+2025%2E+Leveraging+large+language+models+for+code+translation+and+software+development+in+scientific+computing%2E+In+Proceedings+of+the+Platform+for+Advanced+Scientific+Comput)
11. Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and 1 others. 2023. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion. In Advances in Neural Information Processing S… [find](https://scholar.google.com/scholar?q=Yangruibo+Ding%2C+Zijian+Wang%2C+Wasi+Ahmad%2C+Hantian+Ding%2C+Ming+Tan%2C+Nihal+Jain%2C+Murali+Krishna+Ramanathan%2C+Ramesh+Nallapati%2C+Parminder+Bhatia%2C+Dan+Roth%2C+and+1+others%2E+2023%2E+CrossCodeEval%3A+A+Diverse+and+M)
12. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and 1 others. 2020. Code-BERT: A Pre-Trained Model for Programming and Wenchao Gu, Zongyi Lyu, Yanlin Wang, Hongyu Zhang, Cuiyun Gao, and Michael R Lyu. 2025a. SPENCER: Self-Adaptiv… — [arXiv:2508.00546](https://arxiv.org/abs/2508.00546) [find](https://scholar.google.com/scholar?q=Zhangyin+Feng%2C+Daya+Guo%2C+Duyu+Tang%2C+Nan+Duan%2C+Xiaocheng+Feng%2C+Ming+Gong%2C+Linjun+Shou%2C+Bing+Qin%2C+Ting+Liu%2C+Daxin+Jiang%2C+and+1+others%2E+2020%2E+Code%2DBERT%3A+A+Pre%2DTrained+Model+for+Programming+and+Wenchao+Gu)
13. Wenchao Gu, Ensheng Shi, Yanlin Wang, Lun Du, Shi Han, Hongyu Zhang, Dongmei Zhang, and Michael R Lyu. 2025b. SECRET: Towards Scalable and Efficient Code Retrieval via Segmented Deep Hashing. In Proceedings of the IEEE/ACM 47th International Conference on Software Engineering, pages 2303–2315. IEEE. [find](https://scholar.google.com/scholar?q=Wenchao+Gu%2C+Ensheng+Shi%2C+Yanlin+Wang%2C+Lun+Du%2C+Shi+Han%2C+Hongyu+Zhang%2C+Dongmei+Zhang%2C+and+Michael+R+Lyu%2E+2025b%2E+SECRET%3A+Towards+Scalable+and+Efficient+Code+Retrieval+via+Segmented+Deep+Hashing%2E+In+Proce)
14. Wenchao Gu, Yanlin Wang, Lun Du, Hongyu Zhang, Shi Han, Dongmei Zhang, and Michael Lyu. 2022. Accelerating Code Search with Deep Hashing and Code Classification. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 2534–2544. [find](https://scholar.google.com/scholar?q=Wenchao+Gu%2C+Yanlin+Wang%2C+Lun+Du%2C+Hongyu+Zhang%2C+Shi+Han%2C+Dongmei+Zhang%2C+and+Michael+Lyu%2E+2022%2E+Accelerating+Code+Search+with+Deep+Hashing+and+Code+Classification%2E+In+Proceedings+of+the+60th+Annual+Meet)
15. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://md.research.elamur.ai/free/en/papers/guo-2022-unixcoder-unified-cross-modal-pre/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 7212– 7225.
16. Priyanshu Gupta, Avishree Khare, Yasharth Bajpai, Saikat Chakraborty, Sumit Gulwani, Aditya Kanade, Arjun Radhakrishna, Gustavo Soares, and Ashish Tiwari. 2023. Grace: Language Models Meet Code Edits. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the… [find](https://scholar.google.com/scholar?q=Priyanshu+Gupta%2C+Avishree+Khare%2C+Yasharth+Bajpai%2C+Saikat+Chakraborty%2C+Sumit+Gulwani%2C+Aditya+Kanade%2C+Arjun+Radhakrishna%2C+Gustavo+Soares%2C+and+Ashish+Tiwari%2E+2023%2E+Grace%3A+Language+Models+Meet+Code+Edits%2E)
17. Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, and 1 others. 2024. [OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models](https://md.research.elamur.ai/free/en/papers/huang-2025-opencoder-open-cookbook-top-tier/). In CoRR.
18. GitHub Inc. 2024. GitHub Copilot: Your AI Pair Programmer. Accessed: 2024-10-06. [find](https://scholar.google.com/scholar?q=GitHub+Inc%2E+2024%2E+GitHub+Copilot%3A+Your+AI+Pair+Programmer%2E+Accessed%3A+2024%2D10%2D06%2E)
19. Maliheh Izadi, Jonathan Katzy, Tim Van Dam, Marc Otten, Razvan Mihai Popescu, and Arie Van Deursen. 2024. Language Models for Code Completion: A Practical Evaluation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13. [find](https://scholar.google.com/scholar?q=Maliheh+Izadi%2C+Jonathan+Katzy%2C+Tim+Van+Dam%2C+Marc+Otten%2C+Razvan+Mihai+Popescu%2C+and+Arie+Van+Deursen%2E+2024%2E+Language+Models+for+Code+Completion%3A+A+Practical+Evaluation%2E+In+Proceedings+of+the+IEEE%2FACM+46)
20. Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. [SWE-bench: Can Language Models Resolve Real-world Github Issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations.
21. Denis Kocetkov, Raymond Li, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and 1 others. 2023. The Stack: 3 TB of permissively licensed source code. In Transactions on Machine Learning Research. [find](https://scholar.google.com/scholar?q=Denis+Kocetkov%2C+Raymond+Li%2C+Chenghao+Mou%2C+Yacine+Jernite%2C+Margaret+Mitchell%2C+Carlos+Mu%C3%B1oz+Ferrandis%2C+Sean+Hughes%2C+Thomas+Wolf%2C+Dzmitry+Bahdanau%2C+Leandro+Von+Werra%2C+and+1+others%2E+2023%2E+The+Stack%3A+3+TB+)
22. Cursor Team. 2024. Cursor. Accessed: 2024-10-06. [find](https://scholar.google.com/scholar?q=Cursor+Team%2E+2024%2E+Cursor%2E+Accessed%3A+2024%2D10%2D06%2E)
23. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). In Advances in Neural Information Processing Systems, volum…
24. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696–8708. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+CH+Hoi%2E+2021%2E+CodeT5%3A+Identifier%2Daware+Unified+Pre%2Dtrained+Encoder%2DDecoder+Models+for+Code+Understanding+and+Generation%2E+In+Proceedings+of+the+2021+Confe)
25. Ming Liang, Xiaoheng Xie, Gehao Zhang, Xunjin Zheng, Peng Di, Wei Jiang, Hongwei Chen, Chengpeng Wang, and Gang Fan. 2024. REPOFUSE: Repository-Level Code Completion with Fused Dual Context. In CoRR. [find](https://scholar.google.com/scholar?q=Ming+Liang%2C+Xiaoheng+Xie%2C+Gehao+Zhang%2C+Xunjin+Zheng%2C+Peng+Di%2C+Wei+Jiang%2C+Hongwei+Chen%2C+Chengpeng+Wang%2C+and+Gang+Fan%2E+2024%2E+REPOFUSE%3A+Repository%2DLevel+Code+Completion+with+Fused+Dual+Context%2E+In+CoRR%2E)
26. Shanchao Liang, Spandan Garg, and Roshanak Zilouchian Moghaddam. 2025a. The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason. arXiv preprint arXiv:2506.12286. [find](https://scholar.google.com/scholar?q=Shanchao+Liang%2C+Spandan+Garg%2C+and+Roshanak+Zilouchian+Moghaddam%2E+2025a%2E+The+SWE%2DBench+Illusion%3A+When+State%2Dof%2Dthe%2DArt+LLMs+Remember+Instead+of+Reason%2E+arXiv+preprint+arXiv%3A2506%2E12286%2E)
27. Shanchao Liang, Yiran Hu, Nan Jiang, and Lin Tan. 2025b. Can Language Models Replace Programmers? REPOCOD Says ‘Not Yet’. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shanchao+Liang%2C+Yiran+Hu%2C+Nan+Jiang%2C+and+Lin+Tan%2E+2025b%2E+Can+Language+Models+Replace+Programmers%3F+REPOCOD+Says+%E2%80%98Not+Yet%E2%80%99%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+Computational)
28. Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Tianyang+Liu%2C+Canwen+Xu%2C+and+Julian+McAuley%2E+2024%2E+RepoBench%3A+Benchmarking+Repository%2DLevel+Code+Auto%2DCompletion+Systems%2E+In+The+Twelfth+International+Conference+on+Learning+Representations%2E)
29. Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, and 1 others. 2024. StarCoder 2 and The Stack v2: The Next Generation. arXiv preprint arXiv:2402.19173. [find](https://scholar.google.com/scholar?q=Anton+Lozhkov%2C+Raymond+Li%2C+Loubna+Ben+Allal%2C+Federico+Cassano%2C+Joel+Lamy%2DPoirier%2C+Nouamane+Tazi%2C+Ao+Tang%2C+Dmytro+Pykhtar%2C+Jiawei+Liu%2C+Yuxiang+Wei%2C+and+1+others%2E+2024%2E+StarCoder+2+and+The+Stack+v2%3A+The)
30. Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. 2025. RepoGraph: Enhancing AI Software Engineering with Repository-level Code Graph. In The Thirteenth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Siru+Ouyang%2C+Wenhao+Yu%2C+Kaixin+Ma%2C+Zilin+Xiao%2C+Zhihan+Zhang%2C+Mengzhao+Jia%2C+Jiawei+Han%2C+Hongming+Zhang%2C+and+Dong+Yu%2E+2025%2E+RepoGraph%3A+Enhancing+AI+Software+Engineering+with+Repository%2Dlevel+Code+Graph%2E)
31. Debalina Ghosh Paul, Hong Zhu, and Ian Bayley. 2024. Benchmarks and Metrics for Evaluations of Code Generation: A Critical Review. In 2024 IEEE International Conference on Artificial Intelligence Testing (AITest), pages 87–94. IEEE. [find](https://scholar.google.com/scholar?q=Debalina+Ghosh+Paul%2C+Hong+Zhu%2C+and+Ian+Bayley%2E+2024%2E+Benchmarks+and+Metrics+for+Evaluations+of+Code+Generation%3A+A+Critical+Review%2E+In+2024+IEEE+International+Conference+on+Artificial+Intelligence+Test)
32. Stephen Robertson, Hugo Zaragoza, and 1 others. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson%2C+Hugo+Zaragoza%2C+and+1+others%2E+2009%2E+The+Probabilistic+Relevance+Framework%3A+BM25+and+Beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
33. ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, and 1 others. 2025. [Seed-Coder: Let the Code Model Curate Data for Itself](https://md.research.elamur.ai/free/en/papers/bytedance-seed-2025-seed-coder-let-code/). arXiv preprint arXiv:2506.03524.
34. Qwen Team. 2025. Qwen3-Coder: Agentic Coding in the World. Blog post. [find](https://scholar.google.com/scholar?q=Qwen+Team%2E+2025%2E+Qwen3%2DCoder%3A+Agentic+Coding+in+the+World%2E+Blog+post%2E)
35. Zora Zhiruo Wang, Akari Asai, Frank F Xu, Yiqing Xie, Graham Neubig, Daniel Fried, and 1 others. 2025. CodeRAG-Bench: Can Retrieval Augment Code Generation? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3199–3214. [find](https://scholar.google.com/scholar?q=Zora+Zhiruo+Wang%2C+Akari+Asai%2C+Frank+F+Xu%2C+Yiqing+Xie%2C+Graham+Neubig%2C+Daniel+Fried%2C+and+1+others%2E+2025%2E+CodeRAG%2DBench%3A+Can+Retrieval+Augment+Code+Generation%3F+In+Findings+of+the+Association+for+Computat)
36. Michael L. Waskom. 2021. seaborn: statistical data visualization. Journal of Open Source Software, 6(60):3021. [find](https://scholar.google.com/scholar?q=Michael+L%2E+Waskom%2E+2021%2E+seaborn%3A+statistical+data+visualization%2E+Journal+of+Open+Source+Software%2C+6%2860%29%3A3021%2E)
37. Amazon Web Services. 2024. Amazon Q Developer: AI-Powered Coding Assistant. Accessed: 2024-10-06. [find](https://scholar.google.com/scholar?q=Amazon+Web+Services%2E+2024%2E+Amazon+Q+Developer%3A+AI%2DPowered+Coding+Assistant%2E+Accessed%3A+2024%2D10%2D06%2E)
38. Di Wu, Wasi Uddin Ahmad, Dejiao Zhang, Murali Krishna Ramanathan, and Xiaofei Ma. 2024. Repoformer: Selective Retrieval for Repository-Level Code Completion. In Proceedings of the 41st International Conference on Machine Learning, pages 53270–53290. [find](https://scholar.google.com/scholar?q=Di+Wu%2C+Wasi+Uddin+Ahmad%2C+Dejiao+Zhang%2C+Murali+Krishna+Ramanathan%2C+and+Xiaofei+Ma%2E+2024%2E+Repoformer%3A+Selective+Retrieval+for+Repository%2DLevel+Code+Completion%2E+In+Proceedings+of+the+41st+International+C)
39. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. [Qwen3 Technical Report](https://md.research.elamur.ai/free/en/papers/yang-2025-qwen3-technical-report/). arXiv preprint arXiv:2505.09388.
40. Dejiao Zhang, Wasi Uddin Ahmad, Ming Tan, Hantian Ding, Ramesh Nallapati, Dan Roth, Xiaofei Ma, and Bing Xiang. 2024. Code Representation Learning at Scale. In The Twelfth International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Dejiao+Zhang%2C+Wasi+Uddin+Ahmad%2C+Ming+Tan%2C+Hantian+Ding%2C+Ramesh+Nallapati%2C+Dan+Roth%2C+Xiaofei+Ma%2C+and+Bing+Xiang%2E+2024%2E+Code+Representation+Learning+at+Scale%2E+In+The+Twelfth+International+Conference+on+)
41. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pag… [find](https://scholar.google.com/scholar?q=Fengji+Zhang%2C+Bei+Chen%2C+Yue+Zhang%2C+Jacky+Keung%2C+Jin+Liu%2C+Daoguang+Zan%2C+Yi+Mao%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2023%2E+RepoCoder%3A+Repository%2DLevel+Code+Completion+Through+Iterative+Retrieval+and+Genera)
42. Lei Zhang, Yunshui Li, Jiaming Li, Xiaobo Xia, Jiaxi Yang, Run Luo, Minzheng Wang, Longze Chen, Junhao Liu, Qiang Qu, and 1 others. 2025. Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs. In Proceedings of the AAAI Conference on Artificia… [find](https://scholar.google.com/scholar?q=Lei+Zhang%2C+Yunshui+Li%2C+Jiaming+Li%2C+Xiaobo+Xia%2C+Jiaxi+Yang%2C+Run+Luo%2C+Minzheng+Wang%2C+Longze+Chen%2C+Junhao+Liu%2C+Qiang+Qu%2C+and+1+others%2E+2025%2E+Hierarchical+Context+Pruning%3A+Optimizing+Real%2DWorld+Code+Compl)
43. Dewu Zheng, Yanlin Wang, Ensheng Shi, Ruikai Zhang, Yuchi Ma, Hongyu Zhang, and Zibin Zheng. 2025. HumanEvo: An Evolution-Aware Benchmark for More Realistic Evaluation of Repository-Level Code Generation. In 2025 IEEE/ACM 47th International Conference on Software Engineering, pages 1372– 1384. IEEE. [find](https://scholar.google.com/scholar?q=Dewu+Zheng%2C+Yanlin+Wang%2C+Ensheng+Shi%2C+Ruikai+Zhang%2C+Yuchi+Ma%2C+Hongyu+Zhang%2C+and+Zibin+Zheng%2E+2025%2E+HumanEvo%3A+An+Evolution%2DAware+Benchmark+for+More+Realistic+Evaluation+of+Repository%2DLevel+Code+Generat)
44. Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others. 2025. [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://md.research.elamur.ai/free/en/papers/zhuo-2024-bigcodebench-benchmarking-code-generation-diverse/). In The Thirteenth International Conferen…

*Updated 31 October 2026*