---
title: "MultiPL-E — paper by Cassano et al. (2022) — summary and ideas"
description: "Code generation models are trained on many programming languages but are usually tested only on… 44 references, cited by 75 papers here."
authors: ["Federico Cassano", "John Gouwar", "Daniel Nguyen", "Sydney Nguyen", "Luna Phipps-Costin", "Donald Pinckney", "Ming-Ho Yee", "Yangtian Zi", "Carolyn Jane Anderson", "Molly Q Feldman", "Arjun Guha", "Michael Greenberg", "Abhinav Jangda"]
year: 2022
venue: "arXiv:2208.08227"
source: "https://arxiv.org/abs/2208.08227"
doi: "10.48550/arxiv.2208.08227"
arxiv: "2208.08227"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cassano-2022-multipl-e-scalable-extensible-approach/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cassano-2022-multipl-e-scalable-extensible-approach/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

# MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation

**Authors:** [Federico Cassano](https://md.research.elamur.ai/free/en/authors/federico-cassano/), John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, [Carolyn Jane Anderson](https://md.research.elamur.ai/free/en/authors/carolyn-jane-anderson/), Molly Q Feldman, [Arjun Guha](https://md.research.elamur.ai/free/en/authors/arjun-guha/), Michael Greenberg and Abhinav Jangda · **Year:** 2022 · **Venue:** arXiv:2208.08227 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2208.08227) · [PDF](https://arxiv.org/pdf/2208.08227) · [DOI 10.48550/arxiv.2208.08227](https://doi.org/10.48550/arxiv.2208.08227) · [arXiv 2208.08227](https://arxiv.org/abs/2208.08227) · [OpenAlex](https://openalex.org/W4300989059)

**Citations:** 13 citations (OpenAlex, 2026-10-14) · [75](https://md.research.elamur.ai/free/en/papers/cassano-2022-multipl-e-scalable-extensible-approach/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Code generation models are trained on many programming languages but are usually tested only on Python. This leaves open how well they work for developers using other languages. MultiPL-E addresses this by converting Python benchmarks based on unit tests into other languages. It does not translate whole programs. Each target language needs only a small compiler, about 200 lines, that converts four elements: the function signature, the docstring comment, simple test cases, and type annotations where the language requires them. The model writes the function body itself. A rule-based step adjusts Python-specific vocabulary in comments, and a containerized sandbox runs and classifies the generated code. The authors applied this to HumanEval and MBPP, producing parallel versions in 18 additional languages. They then evaluated Codex, CodeGen, and InCoder. Codex performs as well as or better than its Python results in several languages, including JavaScript, C++, Scala, and TypeScript. Because the framework is cheap to extend, it supports fairer multi-language evaluation of code models.

## Key points
- The resources MultiPL-HumanEval and MultiPL-MBPP cover 19 languages in total (Python plus 18 targets) and contain matching problems across languages. They are released at github.com/nuprl/MultiPL-E.
- Adding a language is cheap: a translator of roughly 200 LOC plus an evaluation script of around 20 LOC.
- Across models, JavaScript results are very strong and sometimes surpass Python.
- Model perplexity correlates poorly with functional correctness, so it is an unreliable proxy for code quality.
- Performance generally tracks language popularity, though some niche languages score comparably. Results depend on how prompts are translated, while static typing shows no clear positive or negative effect.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 131 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 168 works

## Cite

Cite as: Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg and Abhinav Jangda. 2022. MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation. arXiv:2208.08227.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{cassano2022multipl,
  title = {{MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation}},
  author = {Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg and Abhinav Jangda},
  year = {2022},
  doi = {10.48550/arxiv.2208.08227},
  eprint = {2208.08227},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2208.08227}
}
```

- [← previous Language Model Cascades](https://md.research.elamur.ai/free/en/papers/dohan-2022-language-model-cascades/)
- [next → The AI teacher test: Measuring the pedagogical ability of Blender and GPT-3 in educational dialogues](https://md.research.elamur.ai/free/en/papers/tack-2022-ai-teacher-test-measuring-pedagogical/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

44 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman et al., “[Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/),” arXiv preprint arXiv:2107.03374, 2021.
2. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton, “[Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/),” arXiv preprint arXiv:2108.07732, 2021. [Online]. Available: https://arxiv.org/abs/2108.07732 [↗](https://arxiv.org/abs/2108.07732)
3. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” 2022. [Online]. Available: https://arxiv.org/abs/2203.13474 [↗](https://arxiv.org/abs/2203.13474)
4. D. Fried, A. Aghajanyan, J. Lin, S. Wang, E. Wallace, F. Shi, R. Zhong, W.-t. Yih, L. Zettlemoyer, and M. Lewis, “Incoder: A generative model for code infilling and synthesis,” arXiv preprint arXiv:2204.05999, 2022. [Online]. Available: https://arxiv.org/abs/2204.05999 [↗](https://arxiv.org/abs/2204.05999)
5. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, M. Pieler, U. S. Prashanth, S. Purohit, L. Reynolds, J. Tow, B. Wang, and S. Weinbach, “[Gpt-neox-20b: An open-source autoregressive language model](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/),” arXiv preprint arXiv:2204.06745, 2022. [Onl…
6. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong, “A conversational paradigm for program synthesis,” arXiv preprint arXiv:2203.13474, 2022. [Online]. Available: https://arxiv.org/abs/2203.13474 [↗](https://arxiv.org/abs/2203.13474)
7. F. F. Xu, U. Alon, G. Neubig, and V. J. Hellendoorn, “[A systematic evaluation of large language models of code](https://md.research.elamur.ai/free/en/papers/xu-2022-systematic-evaluation-large-language-models/),” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 1–10.
8. A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Productivity assessment of neural code completion,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, 2022, pp. 21–29. [find](https://scholar.google.com/scholar?q=A%2E+Ziegler%2C+E%2E+Kalliamvakou%2C+X%2E+A%2E+Li%2C+A%2E+Rice%2C+D%2E+Rifkin%2C+S%2E+Simister%2C+G%2E+Sittampalam%2C+and+E%2E+Aftandilian%2C+%E2%80%9CProductivity+assessment+of+neural+code+completion%2C%E2%80%9D+in+Proceedings+of+the+6th+ACM+SIGPLAN+I)
9. S. Kulal, P. Pasupat, K. Chandra, M. Lee, O. Padon, A. Aiken, and P. S. Liang, “[Spoc: Search-based pseudocode to code](https://md.research.elamur.ai/free/en/papers/kulal-2019-spoc-search-based-pseudocode-code/),” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32. Curran Associates, Inc., 201…
10. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt, “[Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/),” arXiv preprint arXiv:2105.09938, 2021. [Online]. Available: https://arxiv.org/abs/2105.09938 [↗](https://arxiv.org/abs/2105.09938)
11. P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig, “[Learning to mine aligned code and natural language pairs from stack overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/),” in Proceedings of the 15th International Conference on Mining Software Repositories, ser. MSR ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 4…
12. R. Alur, R. Bodik, G. Juniwal, M. Martin, M. Raghothaman, S. A. Seshia, R. Singh, A. Solar-Lezama, E. Torlak, and A. Udupa, “Syntax-guided synthesis,” in Formal Methods in Computer-Aided Design (FMCAD), 2013. [find](https://scholar.google.com/scholar?q=R%2E+Alur%2C+R%2E+Bodik%2C+G%2E+Juniwal%2C+M%2E+Martin%2C+M%2E+Raghothaman%2C+S%2E+A%2E+Seshia%2C+R%2E+Singh%2C+A%2E+Solar%2DLezama%2C+E%2E+Torlak%2C+and+A%2E+Udupa%2C+%E2%80%9CSyntax%2Dguided+synthesis%2C%E2%80%9D+in+Formal+Methods+in+Computer%2DAided+Design+%28FMCAD)
13. S. Chaudhuri, K. Ellis, O. Polozov, R. Singh, A. Solar-Lezama, and Y. Yue, “Neurosymbolic Programming,” Foundations and Trends in Programming Languages, vol. 7, no. 3, pp. 158–243, 2021. [find](https://scholar.google.com/scholar?q=S%2E+Chaudhuri%2C+K%2E+Ellis%2C+O%2E+Polozov%2C+R%2E+Singh%2C+A%2E+Solar%2DLezama%2C+and+Y%2E+Yue%2C+%E2%80%9CNeurosymbolic+Programming%2C%E2%80%9D+Foundations+and+Trends+in+Programming+Languages%2C+vol%2E+7%2C+no%2E+3%2C+pp%2E+158%E2%80%93243%2C+2021%2E)
14. S. Gulwani, O. Polozov, R. Singh et al., “[Program synthesis](https://md.research.elamur.ai/free/en/papers/gulwani-2017-program-synthesis/),” Foundations and Trends® in Programming Languages, vol. 4, no. 1-2, pp. 1–119, 2017.
15. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “[Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/),” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
16. B. Wang and A. Komatsuzaki, “Gpt-j-6b: A 6 billion parameter autoregressive language model,” 2021. [Online]. Available: [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax) [↗](https://github.com/kingoflolz/mesh-transformer-jax)
17. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “[Codebert: A pre-trained model for programming and natural languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/),” arXiv preprint arXiv:2002.08155, 2020. [Online]. Available: https://arxiv.org/abs/2002.08155 [↗](https://arxiv.org/abs/2002.08155)
18. C. Clement, D. Drain, J. Timcheck, A. Svyatkovskiy, and N. Sundaresan, “[PyMT5: multi-mode translation of natural language and python code with transformers](https://md.research.elamur.ai/free/en/papers/clement-2020-pymt5-multi-mode-translation-natural/),” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Lingu…
19. M. Tufano, D. Drain, A. Svyatkovskiy, S. K. Deng, and N. Sundaresan, “Unit test case generation with transformers and focal context,” arXiv preprint arXiv:2009.10297, 2020. [Online]. Available: https://arxiv.org/abs/2009.05617 [↗](https://arxiv.org/abs/2009.05617)
20. S. Lu, D. Guo, S. Ren, J. Huang, A. Svyatkovskiy, A. Blanco, C. Clement, D. Drain, D. Jiang, D. Tang, G. Li, L. Zhou, L. Shou, L. Zhou, M. Tufano, M. Gong, M. Zhou, N. Duan, N. Sundaresan, S. K. Deng, S. Fu, and S. Liu, “Codexglue: A machine learning benchmark dataset for code understanding and gene… [find](https://scholar.google.com/scholar?q=S%2E+Lu%2C+D%2E+Guo%2C+S%2E+Ren%2C+J%2E+Huang%2C+A%2E+Svyatkovskiy%2C+A%2E+Blanco%2C+C%2E+Clement%2C+D%2E+Drain%2C+D%2E+Jiang%2C+D%2E+Tang%2C+G%2E+Li%2C+L%2E+Zhou%2C+L%2E+Shou%2C+L%2E+Zhou%2C+M%2E+Tufano%2C+M%2E+Gong%2C+M%2E+Zhou%2C+N%2E+Duan%2C+N%2E+Sundaresan%2C+S%2E+K%2E+Deng%2C)
21. T. Ahmed and P. Devanbu, “Multilingual training for software engineering,” in Proceedings of the 44th International Conference on Software Engineering. ACM, 2022. [find](https://scholar.google.com/scholar?q=T%2E+Ahmed+and+P%2E+Devanbu%2C+%E2%80%9CMultilingual+training+for+software+engineering%2C%E2%80%9D+in+Proceedings+of+the+44th+International+Conference+on+Software+Engineering%2E+ACM%2C+2022%2E)
22. J. Wei, M. Goyal, G. Durrett, and I. Dillig, “LambdaNet: Probabilistic Type Inference using Graph Neural Networks,” in International Conference on Learning Representations (ICLR), 2020. [find](https://scholar.google.com/scholar?q=J%2E+Wei%2C+M%2E+Goyal%2C+G%2E+Durrett%2C+and+I%2E+Dillig%2C+%E2%80%9CLambdaNet%3A+Probabilistic+Type+Inference+using+Graph+Neural+Networks%2C%E2%80%9D+in+International+Conference+on+Learning+Representations+%28ICLR%29%2C+2020%2E)
23. V. J. Hellendoorn, C. Bird, E. T. Barr, and M. Allamanis, “Deep Learning Type Inference,” in Fse, 2018. [find](https://scholar.google.com/scholar?q=V%2E+J%2E+Hellendoorn%2C+C%2E+Bird%2C+E%2E+T%2E+Barr%2C+and+M%2E+Allamanis%2C+%E2%80%9CDeep+Learning+Type+Inference%2C%E2%80%9D+in+Fse%2C+2018%2E)
24. M. Pradel, G. Gousios, J. Liu, and S. Chandra, “Type-Writer: Neural Type Prediction with Search-Based Validation,” in Esecfse, 2020. [find](https://scholar.google.com/scholar?q=M%2E+Pradel%2C+G%2E+Gousios%2C+J%2E+Liu%2C+and+S%2E+Chandra%2C+%E2%80%9CType%2DWriter%3A+Neural+Type+Prediction+with+Search%2DBased+Validation%2C%E2%80%9D+in+Esecfse%2C+2020%2E)
25. I. Drori, S. Zhang, R. Shuttleworth, L. Tang, A. Lu, E. Ke, K. Liu, L. Chen, S. Tran, N. Cheng, R. Wang, N. Singh, T. L. Patti, J. Lynch, A. Shporer, N. Verma, E. Wu, and G. Strang, “A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning… [find](https://scholar.google.com/scholar?q=I%2E+Drori%2C+S%2E+Zhang%2C+R%2E+Shuttleworth%2C+L%2E+Tang%2C+A%2E+Lu%2C+E%2E+Ke%2C+K%2E+Liu%2C+L%2E+Chen%2C+S%2E+Tran%2C+N%2E+Cheng%2C+R%2E+Wang%2C+N%2E+Singh%2C+T%2E+L%2E+Patti%2C+J%2E+Lynch%2C+A%2E+Shporer%2C+N%2E+Verma%2C+E%2E+Wu%2C+and+G%2E+Strang%2C+%E2%80%9CA+neural+network+)
26. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “[The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/),” in ICLR, 2020.
27. S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” 2020. [Online]. Available: https://arxiv.org/abs/2009.10297 [↗](https://arxiv.org/abs/2009.10297)
28. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “[Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/),” in Proceedings of the 2018 Conference on Empirical Methods in Natural Langu…
29. V. Zhong, C. Xiong, and R. Socher, “[Seq2sql: Generating structured queries from natural language using reinforcement learning](https://md.research.elamur.ai/free/en/papers/zhong-2017-seq2sql-generating-structured-queries-natural/),” 2017. [Online]. Available: https://arxiv.org/abs/1709.00103 [↗](https://arxiv.org/abs/1709.00103)
30. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “[Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/),” in Advances in Neural Information Processing Systems, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curra…
31. A. Aghajanyan, B. Huang, C. Ross, V. Karpukhin, H. Xu, N. Goyal, D. Okhonko, M. Joshi, G. Ghosh, M. Lewis, and L. Zettlemoyer, “Cm3: A causal masked multimodal model of the internet,” arXiv preprint arXiv:2201.07520, 2022. [Online]. Available: https://arxiv.org/abs/2201.07520 [↗](https://arxiv.org/abs/2201.07520)
32. L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, S. Presser, and C. Leahy, “The pile: An 800gb dataset of diverse text for language modeling,” 2021. [Online]. Available: https://arxiv.org/abs/2101.00027 [↗](https://arxiv.org/abs/2101.00027)
33. W. Zaremba, G. Brockman, and OpenAI, “Openai codex,” 2021. [Online]. Available: [https://openai.com/blog/openai-codex/](https://openai.com/blog/openai-codex/) [↗](https://openai.com/blog/openai-codex/)
34. D. Bates, M. Mächler, B. Bolker, and S. Walker, “[Fitting linear mixed-effects models using lme4](https://md.research.elamur.ai/free/en/papers/bates-2015-fitting-linear-mixed-effects-models/),” Journal of Statistical Software, vol. 67, no. 1, pp. 1–48, 2015.
35. B. Chen, F. Zhang, A. Nguyen, D. Zan, Z. Lin, J.-G. Lou, and W. Chen, “CodeT: Code generation with generated tests,” 2022. [Online]. Available: https://arxiv.org/abs/2207.10397 [↗](https://arxiv.org/abs/2207.10397)
36. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “[Bleu: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/),” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ser. ACL ’02. USA: Association for Computational Linguistics, 2002, p. 311–318. [Online]. Availabl…
37. Z. Wang, G. Cuenca, S. Zhou, F. F. Xu, and G. Neubig, “Mconala: A benchmark for code generation from multiple natural languages,” 2022. [Online]. Available: https://arxiv.org/abs/2203.08388 [↗](https://arxiv.org/abs/2203.08388)
38. S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “[Mapping language to code in programmatic context](https://md.research.elamur.ai/free/en/papers/iyer-2018-mapping-language-code-programmatic-context/),” arXiv preprint arXiv:1808.09588, 2018.
39. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “[Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/),” 2018. [Online]. Available: https://arxiv.org/abs/1809.08887 [↗](https://arxiv.org/abs/1809.08887)
40. L. Tunstall, L. von Werra, and T. Wolf, Natural Language Processing with Transformers. O’Reilly Media, 2022. [Online]. Available: [https://books.google.com/books?id=nTxbEAAAQBAJ](https://books.google.com/books?id=nTxbEAAAQBAJ) [↗](https://books.google.com/books?id=nTxbEAAAQBAJ)
41. Y. Li, D. Choi, J. Chung, N. Kushman, J. Schrittwieser, R. Leblond, T. Eccles, J. Keeling, F. Gimeno, A. D. Lago, T. Hubert, P. Choy, C. d. M. d’Autume, I. Babuschkin, X. Chen, P.-S. Huang, J. Welbl, S. Gowal, A. Cherepanov, J. Molloy, D. J. Mankowitz, E. S. Robson, P. Kohli, N. de Freitas, K. Kavuk… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
42. A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
43. B. Athiwaratkun, S. K. Gouda, Z. Wang, X. Li, Y. Tian, M. Tan, W. U. Ahmad, S. Wang, Q. Sun, M. Shang, S. K. Gonugondla, H. Ding, V. Kumar, N. Fulton, A. Farahani, S. Jain, R. Giaquinto, H. Qian, M. K. Ramanathan, R. Nallapati, B. Ray, P. Bhatia, S. Sengupta, D. Roth, and B. Xiang, “Multi-lingual ev… — [Multi-lingual Evaluation of Code Generation Models](https://md.research.elamur.ai/free/en/papers/athiwaratkun-2022-multi-lingual-evaluation-code-generation/)
44. T. Gebru, J. Morgenstern, B. Vecchione, J. W. Vaughan, H. Wallach, H. D. Iii, and K. Crawford, “Datasheets for datasets,” Communications of the ACM, vol. 64, no. 12, pp. 86–92, 2021. [find](https://scholar.google.com/scholar?q=T%2E+Gebru%2C+J%2E+Morgenstern%2C+B%2E+Vecchione%2C+J%2E+W%2E+Vaughan%2C+H%2E+Wallach%2C+H%2E+D%2E+Iii%2C+and+K%2E+Crawford%2C+%E2%80%9CDatasheets+for+datasets%2C%E2%80%9D+Communications+of+the+ACM%2C+vol%2E+64%2C+no%2E+12%2C+pp%2E+86%E2%80%9392%2C+2021%2E)

*Updated 2 November 2026*