---
title: "Execution-Based Evaluation for Open-Domain Code Generation"
description: "The authors present ODEX, a natural-language-to-Python benchmark that applies execution-based… 35 references, cited by 50 papers here."
authors: ["Zhiruo Wang", "Shuyan Zhou", "Daniel Fried", "Graham Neubig"]
year: 2023
venue: "Findings of the Association for Computational Linguistics: EMNLP 2023"
source: "https://aclanthology.org/2023.findings-emnlp.89/"
doi: "10.18653/v1/2023.findings-emnlp.89"
arxiv: "2212.10481"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2023-execution-based-evaluation-open-domain/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2023-execution-based-evaluation-open-domain/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Execution-Based Evaluation for Open-Domain Code Generation

# Execution-Based Evaluation for Open-Domain Code Generation

**Authors:** [Zhiruo Wang](https://md.research.elamur.ai/free/en/authors/zhiruo-wang/), [Shuyan Zhou](https://md.research.elamur.ai/free/en/authors/shuyan-zhou/), [Daniel Fried](https://md.research.elamur.ai/free/en/authors/daniel-fried/) and [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2023 · **Venue:** [Findings of the Association for Computational Linguistics: EMNLP 2023](https://md.research.elamur.ai/free/en/venues/findings/) ([Findings of the Association for Computational Linguistics 2023](https://md.research.elamur.ai/free/en/venues/findings/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.findings-emnlp.89/) · [PDF](https://aclanthology.org/2023.findings-emnlp.89.pdf) · [DOI 10.18653/v1/2023.findings-emnlp.89](https://doi.org/10.18653/v1/2023.findings-emnlp.89) · [arXiv 2212.10481](https://arxiv.org/abs/2212.10481) · [OpenAlex](https://openalex.org/W4389519979)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [50](https://md.research.elamur.ai/free/en/papers/wang-2023-execution-based-evaluation-open-domain/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Most benchmarks that judge generated Python by running it cover only closed-domain tasks, meaning programs that rely on built-in functions or a narrow set of data-science libraries. Real developer requests touch a much wider variety of libraries. The authors present ODEX, a natural-language-to-Python benchmark that applies execution-based checking to these open-domain queries. They took StackOverflow-derived NL-Code pairs from CoNaLa and MCoNaLa and turned each snippet into a self-contained function with explicit library imports. Hired annotators then wrote unit tests, and each test was confirmed against the reference solution. Open-domain code needed special test designs: mocking calls that cannot be reproduced safely, such as HTTP requests; range checks for random outputs; and library-specific equality functions like `np.array_equal`. The authors evaluated Codex and CodeGen. Both models scored clearly worse on library-dependent queries than on closed-domain ones, but both handled non-English intents reasonably well. ODEX matters because it gives code models a more realistic, multilingual testbed and reveals behaviors that benchmarks limited to built-in functions do not show.

## Key points
- **Dataset size and sources:**
  - ODEX contains 945 NL-Code pairs with 1,707 human-written test cases and covers 79 libraries.
  - 53.4% of the problems use at least one library.
- **Intent languages:** the intents are in four languages: English (439), Spanish (90), Japanese (164) and Russian (252).
- **Annotation process:**
  - Four annotators were selected from 20 applicants through a resume screen and a trial task.
  - In the final automatic checks, every canonical solution passed all of its tests (100%).
- **Model findings:**
  - Codex achieves the best overall results.
  - CodeGen scales more efficiently: CodeGen 6.1B performs about as well as Codex 12B.
  - The gap between open-domain and closed-domain performance narrows as CodeGen grows larger, but widens for Codex.
- **Evaluation insights:**
  - Execution-based scoring credits valid alternative solutions that text-match metrics miss.
  - Lexical metrics can still help detect simple bug fixes.
  - Running a single random test per problem gives accuracy close to running the full test set.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

🧭 Topics (2): [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,341 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works

## Cite

Cite as: Zhiruo Wang, Shuyan Zhou, Daniel Fried and Graham Neubig. 2023. Execution-Based Evaluation for Open-Domain Code Generation. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2023execution,
  title = {{Execution-Based Evaluation for Open-Domain Code Generation}},
  author = {Zhiruo Wang, Shuyan Zhou, Daniel Fried and Graham Neubig},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.89},
  eprint = {2212.10481},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.89}
}
```

- [← previous CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation](https://md.research.elamur.ai/free/en/papers/ke-2023-critiquellm-informative-critique-generation/)
- [next → NormBank: A Knowledge Bank of Situational Social Norms](https://md.research.elamur.ai/free/en/papers/ziems-2023-normbank-knowledge-bank-situational-social/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

35 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. 2019. [Juice: A large scale distantly supervised dataset for open domain context-based code generation](https://md.research.elamur.ai/free/en/papers/agashe-2019-juice-large-scale-distantly-supervised/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
2. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023. [Santacoder: don’t reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/)
3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/).
4. Satanjeev Banerjee and Alon Lavie. 2005. [METEOR: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass…
5. Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. 2022. Training and evaluating a jupyter notebook data science assistant. [find](https://scholar.google.com/scholar?q=Shubham+Chandel%2C+Colin+B%2E+Clement%2C+Guillermo+Serrato%2C+and+Neel+Sundaresan%2E+2022%2E+Training+and+evaluating+a+jupyter+notebook+data+science+assistant%2E)
6. Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2023. Codet: Code generation with generated tests. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Bei+Chen%2C+Fengji+Zhang%2C+Anh+Nguyen%2C+Daoguang+Zan%2C+Zeqi+Lin%2C+Jian%2DGuang+Lou%2C+and+Weizhu+Chen%2E+2023%2E+Codet%3A+Code+generation+with+generated+tests%2E+In+International+Conference+on+Learning+Representations%2E)
7. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
8. Naihao Deng, Shuaichen Chang, Peng Shi, Tao Yu, and Rui Zhang. 2021. Prefix-to-sql: Text-to-sql generation from incomplete user questions. [find](https://scholar.google.com/scholar?q=Naihao+Deng%2C+Shuaichen+Chang%2C+Peng+Shi%2C+Tao+Yu%2C+and+Rui+Zhang%2E+2021%2E+Prefix%2Dto%2Dsql%3A+Text%2Dto%2Dsql+generation+from+incomplete+user+questions%2E)
9. Li Dong and Mirella Lapata. 2016. [Language to logical form with neural attention](https://md.research.elamur.ai/free/en/papers/dong-2016-language-logical-form-neural-attention/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33–43, Berlin, Germany. Association for Computational Linguistics.
10. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1383–1392, Melbourne, A… [find](https://scholar.google.com/scholar?q=Rotem+Dror%2C+Gili+Baumer%2C+Segev+Shlomov%2C+and+Roi+Reichart%2E+2018%2E+The+hitchhiker%E2%80%99s+guide+to+testing+statistical+significance+in+natural+language+processing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+)
11. Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2023. Out of the bleu: how should we assess quality of the code generation models? Journal of Systems and Software, 203:111741. [find](https://scholar.google.com/scholar?q=Mikhail+Evtikhiev%2C+Egor+Bogomolov%2C+Yaroslav+Sokolov%2C+and+Timofey+Bryksin%2E+2023%2E+Out+of+the+bleu%3A+how+should+we+assess+quality+of+the+code+generation+models%3F+Journal+of+Systems+and+Software%2C+203%3A111741)
12. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. [find](https://scholar.google.com/scholar?q=Leo+Gao%2C+Stella+Biderman%2C+Sid+Black%2C+Laurence+Golding%2C+Travis+Hoppe%2C+Charles+Foster%2C+Jason+Phang%2C+Horace+He%2C+Anish+Thite%2C+Noa+Nabeshima%2C+Shawn+Presser%2C+and+Connor+Leahy%2E+2020%2E+The+pile%3A+An+800gb+datas)
13. Patrick Haluptzok, Matthew Bowers, and Adam Tauman Kalai. 2023. Language models can teach themselves to program better. [find](https://scholar.google.com/scholar?q=Patrick+Haluptzok%2C+Matthew+Bowers%2C+and+Adam+Tauman+Kalai%2E+2023%2E+Language+models+can+teach+themselves+to+program+better%2E)
14. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/).
15. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/).
16. Junjie Huang, Chenglong Wang, Jipeng Zhang, Cong Yan, Haotian Cui, Jeevana Priya Inala, Colin Clement, and Nan Duan. 2022. Execution-based evaluation for data science code generation models. In Proceedings of the Fourth Workshop on Data Science with Human-in-the-Loop (Language Advances), pages 28–36… [find](https://scholar.google.com/scholar?q=Junjie+Huang%2C+Chenglong+Wang%2C+Jipeng+Zhang%2C+Cong+Yan%2C+Haotian+Cui%2C+Jeevana+Priya+Inala%2C+Colin+Clement%2C+and+Nan+Duan%2E+2022%2E+Execution%2Dbased+evaluation+for+data+science+code+generation+models%2E+In+Procee)
17. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. [Mapping language to code in programmatic context](https://md.research.elamur.ai/free/en/papers/iyer-2018-mapping-language-code-programmatic-context/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643–1652, Brussels, Belgium. Association for Computational Linguistics.
18. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2022. Ds-1000: A natural and reliable benchmark for data science code generation. [find](https://scholar.google.com/scholar?q=Yuhang+Lai%2C+Chengxi+Li%2C+Yiming+Wang%2C+Tianyi+Zhang%2C+Ruiqi+Zhong%2C+Luke+Zettlemoyer%2C+Scott+Wen+tau+Yih%2C+Daniel+Fried%2C+Sida+Wang%2C+and+Tao+Yu%2E+2022%2E+Ds%2D1000%3A+A+natural+and+reliable+benchmark+for+data+scien)
19. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+et+al%2E+2023%2E+Starcoder%3A+may+the+source+be+with+you%21)
20. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). Science, 378(6624):1092–1097.
21. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
22. Chin-Yew Lin and Franz Josef Och. 2004. [Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/). In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 605–612, Barcelona, Spain.
23. Stephan Lukasczyk and Gordon Fraser. 2022. Pyn-guin: Automated unit test generation for python. In International Conference on Software Engineering: Companion Proceedings. [find](https://scholar.google.com/scholar?q=Stephan+Lukasczyk+and+Gordon+Fraser%2E+2022%2E+Pyn%2Dguin%3A+Automated+unit+test+generation+for+python%2E+In+International+Conference+on+Software+Engineering%3A+Companion+Proceedings%2E)
24. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2023. Codegen: An open large language model for code with multi-turn program synthesis. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2023%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+In+)
25. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
26. Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on NLP models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 139–150, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Eric+Wallace%2C+Tony+Zhao%2C+Shi+Feng%2C+and+Sameer+Singh%2E+2021%2E+Concealed+data+poisoning+attacks+on+NLP+models%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Co)
27. Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. 2023. MCoNaLa: A benchmark for code generation from multiple natural languages. In Findings of the Association for Computational Linguistics: EACL 2023, pages 265–273, Dubrovnik, Croatia. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Zhiruo+Wang%2C+Grace+Cuenca%2C+Shuyan+Zhou%2C+Frank+F%2E+Xu%2C+and+Graham+Neubig%2E+2023%2E+MCoNaLa%3A+A+benchmark+for+code+generation+from+multiple+natural+languages%2E+In+Findings+of+the+Association+for+Computational)
28. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. [Learning to mine aligned code and natural language pairs from stack overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/). In International Conference on Mining Software Repositories, pages 476–486. IEEE.
29. Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 7–12, Brussels, Belgium. Association for Co… [find](https://scholar.google.com/scholar?q=Pengcheng+Yin+and+Graham+Neubig%2E+2018%2E+TRANX%3A+A+transition%2Dbased+neural+abstract+syntax+parser+for+semantic+parsing+and+code+generation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+N)
30. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. [Seq2sql: Generating structured queries from natural language using reinforcement learning](https://md.research.elamur.ai/free/en/papers/zhong-2017-seq2sql-generating-structured-queries-natural/).
31. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2021. Asleep at the keyboard? assessing the security of github copilot’s code contributions. [find](https://scholar.google.com/scholar?q=Hammond+Pearce%2C+Baleegh+Ahmad%2C+Benjamin+Tan%2C+Brendan+Dolan%2DGavitt%2C+and+Ramesh+Karri%2E+2021%2E+Asleep+at+the+keyboard%3F+assessing+the+security+of+github+copilot%E2%80%99s+code+contributions%2E)
32. Maja Popović. 2015. [chrF: character n-gram F-score for automatic MT evaluation](https://md.research.elamur.ai/free/en/papers/popovic-2015-chrf-character-n-gram-f/). In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
33. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. Codebleu: a method for automatic evaluation of code synthesis. [find](https://scholar.google.com/scholar?q=Shuo+Ren%2C+Daya+Guo%2C+Shuai+Lu%2C+Long+Zhou%2C+Shujie+Liu%2C+Duyu+Tang%2C+Neel+Sundaresan%2C+Ming+Zhou%2C+Ambrosio+Blanco%2C+and+Shuai+Ma%2E+2020%2E+Codebleu%3A+a+method+for+automatic+evaluation+of+code+synthesis%2E)
34. Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit test case generation with transformers and focal context. [find](https://scholar.google.com/scholar?q=Michele+Tufano%2C+Dawn+Drain%2C+Alexey+Svyatkovskiy%2C+Shao+Kun+Deng%2C+and+Neel+Sundaresan%2E+2021%2E+Unit+test+case+generation+with+transformers+and+focal+context%2E)
35. Morteza Verdi, Ashkan Sami, Jafar Akhondali, Foutse Khomh, Gias Uddin, and Alireza Karami Motlagh. 2020. An empirical study of c++ vulnerabilities in crowd-sourced code examples. IEEE Transactions on Software Engineering. [find](https://scholar.google.com/scholar?q=Morteza+Verdi%2C+Ashkan+Sami%2C+Jafar+Akhondali%2C+Foutse+Khomh%2C+Gias+Uddin%2C+and+Alireza+Karami+Motlagh%2E+2020%2E+An+empirical+study+of+c%2B%2B+vulnerabilities+in+crowd%2Dsourced+code+examples%2E+IEEE+Transactions+on+)

*Updated 2 November 2026*