---
title: "MultiFileTest — paper by Wang et al. (2026) — summary and ideas"
description: "Most benchmarks for LLM-based unit test generation work at the level of a single function, class, or file. 43 references."
authors: ["Yibo Wang", "Congying Xia", "Wenting Zhao", "Jiangshu Du", "Chunyu Miao", "Zhongfen Deng", "Philip S. Yu", "Chen Xing"]
year: 2026
venue: "Findings of the Association for Computational Linguistics: ACL 2026"
source: "https://aclanthology.org/2026.findings-acl.1403/"
doi: "10.18653/v1/2026.findings-acl.1403"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2026-multifiletest-multi-file-level-llm/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2026-multifiletest-multi-file-level-llm/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/) / MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms

# MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms

**Authors:** [Yibo Wang](https://md.research.elamur.ai/free/en/authors/yibo-wang/), [Congying Xia](https://md.research.elamur.ai/free/en/authors/congying-xia/), [Wenting Zhao](https://md.research.elamur.ai/free/en/authors/wenting-zhao/), [Jiangshu Du](https://md.research.elamur.ai/free/en/authors/jiangshu-du/), [Chunyu Miao](https://md.research.elamur.ai/free/en/authors/chunyu-miao/), [Zhongfen Deng](https://md.research.elamur.ai/free/en/authors/zhongfen-deng/), [Philip S. Yu](https://md.research.elamur.ai/free/en/authors/philip-s-yu/) and [Chen Xing](https://md.research.elamur.ai/free/en/authors/chen-xing/) · **Year:** 2026 · **Venue:** [Findings of the Association for Computational Linguistics: ACL 2026](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2026](https://md.research.elamur.ai/free/en/venues/findings/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2026.findings-acl.1403/) · [PDF](https://aclanthology.org/2026.findings-acl.1403.pdf) · [DOI 10.18653/v1/2026.findings-acl.1403](https://doi.org/10.18653/v1/2026.findings-acl.1403) · [OpenAlex](https://openalex.org/W4407359146)

**Citations:** 2 citations (OpenAlex, 2026-10-14)

## Summary
Most benchmarks for LLM-based unit test generation work at the level of a single function, class, or file. Real software usually spreads logic across many files that depend on one another. To close this gap, the authors built MultiFileTest, a benchmark of 60 self-contained GitHub projects: 20 each in Python, Java, and JavaScript. Every project has real cross-file dependencies and is small enough to fit in a model's context window. They tested eleven frontier LLMs under three setups:
- generating tests directly,
- generating tests after humans manually repair executability and cascade errors,
- letting the model repair its own errors.

Tests were scored on executability, correctness, and coverage. Overall performance was only moderate. Gemini 3.0 Pro was strongest overall, and Java proved the hardest language. Even top models made basic mistakes, such as wrong imports, hallucinated APIs, and faulty mocking. Model rankings changed noticeably once errors were fixed, and self-repair fell short of human repair. The work shows that understanding dependencies between files is still a central weakness of LLM test generators.

## Key points
- **Benchmark scope:** 60 projects (20 per language), each with 2–15 files and under 1,600 LOC. This is 3.75× the 16 multi-file unit-testing targets in DevBench.
- **Project size by language:**

  | Language | Avg. files | Avg. LOC |
  |---|---|---|
  | Python | 6.10 | 654.60 |
  | Java | 4.79 | 292.26 |
  | JavaScript | 4.00 | 558.05 |

- **Metrics:** executability rate (ER), correctness rate (CR), and line coverage.
- **Rankings:** Gemini 3.0 Pro achieved the top results for both Python and Java, whereas the best JavaScript performance came from OpenAI o1. Java was the most difficult language, mainly because of its stricter syntax.
- **Common errors:** the most frequent failure after fixing was a mismatch between expected and actual values. Hallucinated functions or classes and missing required code persisted across all three languages.
- **Agentic case study:** a Cursor Agent run with GPT-5 Mini on four projects gave mixed results. It still failed on bankingApplication after 10 repair rounds.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 56 works · 17,867 citations
- [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 61 works · 20,107 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [💻 Software](https://md.research.elamur.ai/free/en/shelves/computer-science/software/) · 92 works · [🌐 Computer Networks and Communications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-networks-and-communications/) · 64 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S. Yu and Chen Xing. 2026. MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2026multifiletest,
  title = {{MultiFileTest: A Multi-File-Level LLM Unit Test Generation Benchmark and Impact of Error Fixing Mechanisms}},
  author = {Yibo Wang, Congying Xia, Wenting Zhao, Jiangshu Du, Chunyu Miao, Zhongfen Deng, Philip S. Yu and Chen Xing},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1403},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1403}
}
```

- [← previous Multi-Granularity Semantic Revision for Large Language Model Distillation](https://md.research.elamur.ai/free/en/papers/liu-2026-multi-granularity-semantic-revision-large/)
- [next → NavA³: Understanding Any Instruction, Navigating Anywhere, Finding Anything](https://md.research.elamur.ai/free/en/papers/zhang-2026-nava-3-understanding-any-instruction/)
- [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/8/)

## References

43 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/). arXiv preprint arXiv:2303.08774.
2. Saranya Alagarsamy, Chakkrit Tantithamthavorn, Wannita Takerngsaksiri, Chetan Arora, and Aldeida Aleti. 2025. Enhancing large language models for text-to-testcase generation. Journal of Systems and Software, page 112531. [find](https://scholar.google.com/scholar?q=Saranya+Alagarsamy%2C+Chakkrit+Tantithamthavorn%2C+Wannita+Takerngsaksiri%2C+Chetan+Arora%2C+and+Aldeida+Aleti%2E+2025%2E+Enhancing+large+language+models+for+text%2Dto%2Dtestcase+generation%2E+Journal+of+Systems+and+So)
3. M Moein Almasi, Hadi Hemmati, Gordon Fraser, Andrea Arcuri, and Janis Benefelds. 2017. An industrial evaluation of unit test generation: Finding real faults in a financial application. In 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICS… [find](https://scholar.google.com/scholar?q=M+Moein+Almasi%2C+Hadi+Hemmati%2C+Gordon+Fraser%2C+Andrea+Arcuri%2C+and+Janis+Benefelds%2E+2017%2E+An+industrial+evaluation+of+unit+test+generation%3A+Finding+real+faults+in+a+financial+application%2E+In+2017+IEEE%2FAC)
4. Amr Almorsi, Mohanned Ahmed, and Walid Gomaa. 2024. Guided code generation with llms: A multi-agent framework for complex code tasks. In 2024 12th International Japan-Africa Conference on Electronics, Communications, and Computations (JAC-ECC), pages 215–218. IEEE. [find](https://scholar.google.com/scholar?q=Amr+Almorsi%2C+Mohanned+Ahmed%2C+and+Walid+Gomaa%2E+2024%2E+Guided+code+generation+with+llms%3A+A+multi%2Dagent+framework+for+complex+code+tasks%2E+In+2024+12th+International+Japan%2DAfrica+Conference+on+Electronics%2C)
5. AI Anthropic. 2024. Claude 3.5 sonnet model card addendum. Claude-3.5 Model Card, 3:6. [find](https://scholar.google.com/scholar?q=AI+Anthropic%2E+2024%2E+Claude+3%2E5+sonnet+model+card+addendum%2E+Claude%2D3%2E5+Model+Card%2C+3%3A6%2E)
6. Mihir Athale and Vishal Vaddina. 2025. Knowledge graph based repository-level code generation. In 2025 IEEE/ACM International Workshop on Large Language Models for Code (LLM4Code), pages 169– 176. IEEE. [find](https://scholar.google.com/scholar?q=Mihir+Athale+and+Vishal+Vaddina%2E+2025%2E+Knowledge+graph+based+repository%2Dlevel+code+generation%2E+In+2025+IEEE%2FACM+International+Workshop+on+Large+Language+Models+for+Code+%28LLM4Code%29%2C+pages+169%E2%80%93+176%2E+IEE)
7. Ermira Daka and Gordon Fraser. 2014. A survey on unit testing practices and problems. In 2014 IEEE 25th International Symposium on Software Reliability Engineering, pages 201–211. IEEE. [find](https://scholar.google.com/scholar?q=Ermira+Daka+and+Gordon+Fraser%2E+2014%2E+A+survey+on+unit+testing+practices+and+problems%2E+In+2014+IEEE+25th+International+Symposium+on+Software+Reliability+Engineering%2C+pages+201%E2%80%93211%2E+IEEE%2E)
8. Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C Desmarais. 2024. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology, 171:107468. [find](https://scholar.google.com/scholar?q=Arghavan+Moradi+Dakhel%2C+Amin+Nikanjam%2C+Vahid+Majdinasab%2C+Foutse+Khomh%2C+and+Michel+C+Desmarais%2E+2024%2E+Effective+test+generation+using+pre%2Dtrained+large+language+models+and+mutation+testing%2E+Information)
9. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. [Longrope: Extending llm context window beyond 2 million tokens](https://md.research.elamur.ai/free/en/papers/ding-2024-longrope-extending-llm-context/). In Forty-first International Conference on Machine Learning.
10. Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2023. Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv e-prints, pages arXiv–2308. [find](https://scholar.google.com/scholar?q=Xueying+Du%2C+Mingwei+Liu%2C+Kaixin+Wang%2C+Hanlin+Wang%2C+Junwei+Liu%2C+Yixuan+Chen%2C+Jiayi+Feng%2C+Chaofeng+Sha%2C+Xin+Peng%2C+and+Yiling+Lou%2E+2023%2E+Classeval%3A+A+manually%2Dcrafted+benchmark+for+evaluating+llms+on+cla)
11. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code-BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547,… [find](https://scholar.google.com/scholar?q=Zhangyin+Feng%2C+Daya+Guo%2C+Duyu+Tang%2C+Nan+Duan%2C+Xiaocheng+Feng%2C+Ming+Gong%2C+Linjun+Shou%2C+Bing+Qin%2C+Ting+Liu%2C+Daxin+Jiang%2C+and+Ming+Zhou%2E+2020%2E+Code%2DBERT%3A+A+pre%2Dtrained+model+for+programming+and+natural+l)
12. Gordon Fraser and Andrea Arcuri. 2011. Evosuite: automatic test suite generation for object-oriented software. In Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering, pages 416–419. [find](https://scholar.google.com/scholar?q=Gordon+Fraser+and+Andrea+Arcuri%2E+2011%2E+Evosuite%3A+automatic+test+suite+generation+for+object%2Doriented+software%2E+In+Proceedings+of+the+19th+ACM+SIGSOFT+symposium+and+the+13th+European+conference+on+Foun)
13. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. [Qwen technical report](https://md.research.elamur.ai/free/en/papers/bai-2023-qwen-technical-report/). arXiv preprint arXiv:2309.16609.
14. Giovanni Grano, Fabio Palomba, Dario Di Nucci, Andrea De Lucia, and Harald C Gall. 2019. Scented since the beginning: On the diffuseness of test smells in automatically generated test code. Journal of Systems and Software, 156:312–327. [find](https://scholar.google.com/scholar?q=Giovanni+Grano%2C+Fabio+Palomba%2C+Dario+Di+Nucci%2C+Andrea+De+Lucia%2C+and+Harald+C+Gall%2E+2019%2E+Scented+since+the+beginning%3A+On+the+diffuseness+of+test+smells+in+automatically+generated+test+code%2E+Journal+of)
15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
16. Giovanni Grano, Simone Scalabrino, Harald C Gall, and Rocco Oliveto. 2018. An empirical investigation on the readability of manual and generated test cases. In Proceedings of the 26th Conference on Program Comprehension, pages 348–351. [find](https://scholar.google.com/scholar?q=Giovanni+Grano%2C+Simone+Scalabrino%2C+Harald+C+Gall%2C+and+Rocco+Oliveto%2E+2018%2E+An+empirical+investigation+on+the+readability+of+manual+and+generated+test+cases%2E+In+Proceedings+of+the+26th+Conference+on+Pr)
17. Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pages 572–576. [find](https://scholar.google.com/scholar?q=Yinghao+Chen%2C+Zehao+Hu%2C+Chen+Zhi%2C+Junxiao+Han%2C+Shuiguang+Deng%2C+and+Jianwei+Yin%2E+2024%2E+Chatunitest%3A+A+framework+for+llm%2Dbased+test+generation%2E+In+Companion+Proceedings+of+the+32nd+ACM+International+Con)
18. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. [find](https://scholar.google.com/scholar?q=Gheorghe+Comanici%2C+Eric+Bieber%2C+Mike+Schaekermann%2C+Ice+Pasupat%2C+Noveen+Sachdeva%2C+Inderjit+Dhillon%2C+Marcel+Blistein%2C+Ori+Ram%2C+Dan+Zhang%2C+Evan+Rosen%2C+et+al%2E)
19. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv e-prints, pages arXiv– 2401. [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Qihao+Zhu%2C+Dejian+Yang%2C+Zhenda+Xie%2C+Kai+Dong%2C+Wentao+Zhang%2C+Guanting+Chen%2C+Xiao+Bi%2C+Y+Wu%2C+YK+Li%2C+et+al%2E+2024%2E+Deepseek%2Dcoder%3A+When+the+large+language+model+meets+programming%E2%80%93the+rise+of+code)
20. Mark Harman and Phil McMinn. 2009. A theoretical and empirical study of search-based testing: Local, global, and hybrid search. IEEE Transactions on Software Engineering, 36(2):226–247. [find](https://scholar.google.com/scholar?q=Mark+Harman+and+Phil+McMinn%2E+2009%2E+A+theoretical+and+empirical+study+of+search%2Dbased+testing%3A+Local%2C+global%2C+and+hybrid+search%2E+IEEE+Transactions+on+Software+Engineering%2C+36%282%29%3A226%E2%80%93247%2E)
21. Kush Jain, Gabriel Synnaeve, and Baptiste Rozière. 2024a. Testgeneval: A real world unit test generation and test completion benchmark. arXiv preprint arXiv:2410.00752. — [arXiv:2410.00752](https://arxiv.org/abs/2410.00752) [find](https://scholar.google.com/scholar?q=Kush+Jain%2C+Gabriel+Synnaeve%2C+and+Baptiste+Rozi%C3%A8re%2E+2024a%2E+Testgeneval%3A+A+real+world+unit+test+generation+and+test+completion+benchmark%2E+arXiv+preprint+arXiv%3A2410%2E00752%2E+%E2%80%94+%5BarXiv%3A2410%2E00752%5D%28https%3A%2F%2Far)
22. Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica. 2024b. R2e: Turning any github repository into a programming agent environment. In Forty-first International Conference on Machine Learning. [find](https://scholar.google.com/scholar?q=Naman+Jain%2C+Manish+Shetty%2C+Tianjun+Zhang%2C+King+Han%2C+Koushik+Sen%2C+and+Ion+Stoica%2E+2024b%2E+R2e%3A+Turning+any+github+repository+into+a+programming+agent+environment%2E+In+Forty%2Dfirst+International+Conference)
23. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. [Swe-bench: Can language models resolve real-world github issues?](https://md.research.elamur.ai/free/en/papers/jimenez-2024-swe-bench-can-language-models/) In The Twelfth International Conference on Learning Representations.
24. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. [Retrieval-augmented generation for knowledge-intensive nlp tasks](https://md.research.elamur.ai/free/en/papers/lewis-2020-retrieval-augmented-generation-knowledge-intensive/). Advances in neural information processing systems, 33:9459–9474.
25. Bowen Li, Wenhan Wu, Ziwei Tang, Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, et al. 2024. Devbench: A comprehensive benchmark for software development. arXiv preprint arXiv:2403.08604. — [arXiv:2403.08604](https://arxiv.org/abs/2403.08604) [find](https://scholar.google.com/scholar?q=Bowen+Li%2C+Wenhan+Wu%2C+Ziwei+Tang%2C+Lin+Shi%2C+John+Yang%2C+Jinyang+Li%2C+Shunyu+Yao%2C+Chen+Qian%2C+Binyuan+Hui%2C+Qicheng+Zhang%2C+et+al%2E+2024%2E+Devbench%3A+A+comprehensive+benchmark+for+software+development%2E+arXiv+pre)
26. Tsz-On Li, Wenxi Zong, Yibo Wang, Haoye Tian, Ying Wang, Shing-Chi Cheung, and Jeff Kramer. 2023. Nuances are the key: Unlocking chatgpt to find failure-inducing tests with differential prompting. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 14–26. IE… [find](https://scholar.google.com/scholar?q=Tsz%2DOn+Li%2C+Wenxi+Zong%2C+Yibo+Wang%2C+Haoye+Tian%2C+Ying+Wang%2C+Shing%2DChi+Cheung%2C+and+Jeff+Kramer%2E+2023%2E+Nuances+are+the+key%3A+Unlocking+chatgpt+to+find+failure%2Dinducing+tests+with+differential+prompting%2E+In+)
27. Stephan Lukasczyk and Gordon Fraser. 2022. Pynguin: Automated unit test generation for python. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings, pages 168–172. [find](https://scholar.google.com/scholar?q=Stephan+Lukasczyk+and+Gordon+Fraser%2E+2022%2E+Pynguin%3A+Automated+unit+test+generation+for+python%2E+In+Proceedings+of+the+ACM%2FIEEE+44th+International+Conference+on+Software+Engineering%3A+Companion+Proceedin)
28. Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, and Jie Tang. 2025. Dynamic scaling of unit tests for code reward modeling. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6917–6935, Vienna, Austria. Association for… [find](https://scholar.google.com/scholar?q=Zeyao+Ma%2C+Xiaokang+Zhang%2C+Jing+Zhang%2C+Jifan+Yu%2C+Sijia+Luo%2C+and+Jie+Tang%2E+2025%2E+Dynamic+scaling+of+unit+tests+for+code+reward+modeling%2E+In+Proceedings+of+the+63rd+Annual+Meeting+of+the+Association+for+)
29. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. [Code llama: Open foundation models for code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/). arXiv preprint arXiv:2308.12950.
30. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering. [find](https://scholar.google.com/scholar?q=Max+Sch%C3%A4fer%2C+Sarah+Nadi%2C+Aryaz+Eghbali%2C+and+Frank+Tip%2E+2023%2E+An+empirical+evaluation+of+using+large+language+models+for+automated+unit+test+generation%2E+IEEE+Transactions+on+Software+Engineering%2E)
31. Mohammed Latif Siddiq, Joanna Cecilia Da Silva Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. 2024. Using large language models to generate junit tests: An empirical study. In Proceedings of the 28th International Conference on Evaluation and Assessment in… [find](https://scholar.google.com/scholar?q=Mohammed+Latif+Siddiq%2C+Joanna+Cecilia+Da+Silva+Santos%2C+Ridwanul+Hasan+Tanvir%2C+Noshin+Ulfat%2C+Fahmid+Al+Rifat%2C+and+Vin%C3%ADcius+Carvalho+Lopes%2E+2024%2E+Using+large+language+models+to+generate+junit+tests%3A+An+)
32. CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A Choquette-Choo, Jingyue Shen, Joe Kelley, et al. 2024a. [Codegemma: Open code models based on gemma](https://md.research.elamur.ai/free/en/papers/zhao-2024-codegemma-open-code-models-based/). arXiv preprint arXiv:2406.11409.
33. Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024b. [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://md.research.elamur.ai/free/en/papers/gemini-team-2024-gemini-1-5-unlocking-multimodal/). arXiv preprint arXiv:2403.05530.
34. Johannes Villmow, Jonas Depoix, and Adrian Ulges. 2021. ConTest: A unit test completion benchmark featuring context. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pages 17–25, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Johannes+Villmow%2C+Jonas+Depoix%2C+and+Adrian+Ulges%2E+2021%2E+ConTest%3A+A+unit+test+completion+benchmark+featuring+context%2E+In+Proceedings+of+the+1st+Workshop+on+Natural+Language+Processing+for+Programming+%28)
35. Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2025. Testeval: Benchmarking large language models for test case generation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3547–3562. [find](https://scholar.google.com/scholar?q=Wenhan+Wang%2C+Chenyuan+Yang%2C+Zhijie+Wang%2C+Yuheng+Huang%2C+Zhaoyang+Chu%2C+Da+Song%2C+Lingming+Zhang%2C+An+Ran+Chen%2C+and+Lei+Ma%2E+2025%2E+Testeval%3A+Benchmarking+large+language+models+for+test+case+generation%2E+In+F)
36. AI Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. [https://ai.meta](https://ai.meta). com/blog/llama-4-multimodal-intelligence/, checked on, 4(7):2025. [↗](https://ai.meta)
37. Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. 2023. CodeT5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1069–1088, Singapore. Association for Compu… [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Hung+Le%2C+Akhilesh+Gotmare%2C+Nghi+Bui%2C+Junnan+Li%2C+and+Steven+Hoi%2E+2023%2E+CodeT5%2B%3A+Open+code+large+language+models+for+code+understanding+and+generation%2E+In+Proceedings+of+the+2023+Conference+on)
38. Niels Mündler, Mark Niklas Mueller, Jingxuan He, and Martin Vechev. 2024. Swt-bench: Testing and validating real-world bug-fixes with code agents. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Niels+M%C3%BCndler%2C+Mark+Niklas+Mueller%2C+Jingxuan+He%2C+and+Martin+Vechev%2E+2024%2E+Swt%2Dbench%3A+Testing+and+validating+real%2Dworld+bug%2Dfixes+with+code+agents%2E+In+The+Thirty%2Deighth+Annual+Conference+on+Neural+Info)
39. Zejun Wang, Kaibo Liu, Ge Li, and Zhi Jin. 2024. Hits: High-coverage llm-based unit test generation via method slicing. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, pages 1258–1268. [find](https://scholar.google.com/scholar?q=Zejun+Wang%2C+Kaibo+Liu%2C+Ge+Li%2C+and+Zhi+Jin%2E+2024%2E+Hits%3A+High%2Dcoverage+llm%2Dbased+unit+test+generation+via+method+slicing%2E+In+Proceedings+of+the+39th+IEEE%2FACM+International+Conference+on+Automated+Softwa)
40. Carlos Pacheco, Shuvendu K Lahiri, Michael D Ernst, and Thomas Ball. 2007. Feedback-directed random test generation. In 29th International Conference Xusheng Xiao, Sihan Li, Tao Xie, and Nikolai Tillmann. 2013. Characteristic studies of loop problems for structural test generation via symbolic execu… [find](https://scholar.google.com/scholar?q=Carlos+Pacheco%2C+Shuvendu+K+Lahiri%2C+Michael+D+Ernst%2C+and+Thomas+Ball%2E+2007%2E+Feedback%2Ddirected+random+test+generation%2E+In+29th+International+Conference+Xusheng+Xiao%2C+Sihan+Li%2C+Tao+Xie%2C+and+Nikolai+Tillm)
41. Zhuokui Xie, Yinghao Chen, Chen Zhi, Shuiguang Deng, and Jianwei Yin. 2023. Chatunitest: a chatgpt-based automated unit test generation tool. arXiv preprint arXiv:2305.04764. — [arXiv:2305.04764](https://arxiv.org/abs/2305.04764) [find](https://scholar.google.com/scholar?q=Zhuokui+Xie%2C+Yinghao+Chen%2C+Chen+Zhi%2C+Shuiguang+Deng%2C+and+Jianwei+Yin%2E+2023%2E+Chatunitest%3A+a+chatgpt%2Dbased+automated+unit+test+generation+tool%2E+arXiv+preprint+arXiv%3A2305%2E04764%2E+%E2%80%94+%5BarXiv%3A2305%2E04764%5D%28http)
42. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pag… [find](https://scholar.google.com/scholar?q=Fengji+Zhang%2C+Bei+Chen%2C+Yue+Zhang%2C+Jacky+Keung%2C+Jin+Liu%2C+Daoguang+Zan%2C+Yi+Mao%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2023%2E+Repocoder%3A+Repository%2Dlevel+code+completion+through+iterative+retrieval+and+genera)
43. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al. 2024. [Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence](https://md.research.elamur.ai/free/en/papers/deepseek-ai-2024-deepseek-coder-v2-breaking-barrier/). arXiv preprint arXiv:2406.11931.

*Updated 31 October 2026*