---
title: "PanGu-Coder — paper by Christopoulou et al. (2022)"
description: "The authors present PanGu-Coder, a decoder-only model built on the PanGu-α architecture. 73 references, cited by 26 papers here."
authors: ["Fenia Christopoulou", "Gerasimos Lampouras", "Milan Gritta", "Guchun Zhang", "Yinpeng Guo", "Zhongqi Li", "Qi Zhang", "Meng Xiao", "Bo Shen", "Lin Li", "Hao Yu", "Li Yan", "Pingyi Zhou", "Xin Wang", "Yuchi Ma", "Ignacio Iacobacci", "Yasheng Wang", "Guangtai Liang", "Jiansheng Wei", "Xin Jiang", "Qianxiang Wang", "Qun Liu"]
year: 2022
venue: "arXiv:2207.11280"
source: "https://arxiv.org/abs/2207.11280"
doi: "10.48550/arxiv.2207.11280"
arxiv: "2207.11280"
licence: "arXiv — non-exclusive licence to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/christopoulou-2022-pangu-coder-program-synthesis/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/christopoulou-2022-pangu-coder-program-synthesis/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / PanGu-Coder: Program Synthesis with Function-Level Language Modeling

# PanGu-Coder: Program Synthesis with Function-Level Language Modeling

**Authors:** [Fenia Christopoulou](https://md.research.elamur.ai/free/en/authors/fenia-christopoulou/), Gerasimos Lampouras, [Milan Gritta](https://md.research.elamur.ai/free/en/authors/milan-gritta/), [Guchun Zhang](https://md.research.elamur.ai/free/en/authors/guchun-zhang/), Yinpeng Guo, Zhongqi Li, [Qi Zhang](https://md.research.elamur.ai/free/en/authors/qi-zhang/), Meng Xiao, [Bo Shen](https://md.research.elamur.ai/free/en/authors/bo-shen/), [Lin Li](https://md.research.elamur.ai/free/en/authors/lin-li/), [Hao Yu](https://md.research.elamur.ai/free/en/authors/hao-yu/), Li Yan, Pingyi Zhou, [Xin Wang](https://md.research.elamur.ai/free/en/authors/xin-wang/), Yuchi Ma, [Ignacio Iacobacci](https://md.research.elamur.ai/free/en/authors/ignacio-iacobacci/), [Yasheng Wang](https://md.research.elamur.ai/free/en/authors/yasheng-wang/), Guangtai Liang, [Jiansheng Wei](https://md.research.elamur.ai/free/en/authors/jiansheng-wei/), [Xin Jiang](https://md.research.elamur.ai/free/en/authors/xin-jiang/), Qianxiang Wang and [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/) · **Year:** 2022 · **Venue:** arXiv:2207.11280 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2207.11280) · [PDF](https://arxiv.org/pdf/2207.11280) · [DOI 10.48550/arxiv.2207.11280](https://doi.org/10.48550/arxiv.2207.11280) · [arXiv 2207.11280](https://arxiv.org/abs/2207.11280) · [OpenAlex](https://openalex.org/W4288055861)

**Citations:** 23 citations (OpenAlex, 2026-10-14) · [26](https://md.research.elamur.ai/free/en/papers/christopoulou-2022-pangu-coder-program-synthesis/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper addresses text-to-code generation: producing a Python function body that passes unit tests, given an English problem description and a function signature. The authors present PanGu-Coder, a decoder-only model built on the PanGu-α architecture. That architecture adds a query layer on top of the transformer stack, which uses the embedding of the next position to predict the next token.

Training happens in two stages. First, the model learns from raw Python data, including any docstrings and comments, with standard Causal Language Modelling. Second, it trains only on docstring–function pairs, using mixtures of CLM and MLM-style objectives inspired by encoder-decoder setups. A further variant, PanGu-Coder-FT, is fine-tuned on competitive programming tasks and on code covered by continuous integration tests.

On HumanEval and MBPP, which measure functional correctness, PanGu-Coder matches or beats similarly sized models such as Codex. It does so with a shorter context window and less training data. This suggests that task-focused training on aligned description–code pairs can stand in for some of the extra scale.

## Key points
- Two sizes are trained: 317M parameters (24 layers, hidden size 1,024) and 2.6B parameters (32 layers, hidden size 2,560). Both use a 1,024-token context and a 41,865-token vocabulary.
- The pre-training corpus comes from GitHub repositories created before May 2021, collected via GHTorrent:
  - about 65M Python files (380 GB) at the start;
  - about 40M files (185 GB) after MD5 deduplication;
  - roughly 147 GB after quality filters (file size, Python3 AST validity, line-length limits).
- Functions and docstrings are extracted by AST parsing and formatted with the special tokens `<descr>`, `<python>` and `<eoc>`. Stage 2 objectives include CODE-CLM, DOCSTR-MLM and DOCSTR-MCLM.
- Evaluation is zero-shot on HumanEval (164 problems, about 7.7 hidden tests each) and on MBPP (974 problems).
- PanGu-Coder-FT turns out to be sensitive to shifts in the fine-tuning data distribution. Selecting fine-tuning examples by similarity to a centroid embedding helps, and post-processing that filters out failing programs raises pass rates notably.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works · [🖥️ Computer Science Applications](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-science-applications/) · 102 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, Hao Yu, Li Yan, Pingyi Zhou, Xin Wang, Yuchi Ma, Ignacio Iacobacci, Yasheng Wang, Guangtai Liang, Jiansheng Wei, Xin Jiang, Qianxiang Wang and Qun Liu. 2022. PanGu-Coder: Program Synthesis with Function-Level Language Modeling. arXiv:2207.11280.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{christopoulou2022pangu,
  title = {{PanGu-Coder: Program Synthesis with Function-Level Language Modeling}},
  author = {Fenia Christopoulou, Gerasimos Lampouras, Milan Gritta, Guchun Zhang, Yinpeng Guo, Zhongqi Li, Qi Zhang, Meng Xiao, Bo Shen, Lin Li, Hao Yu, Li Yan, Pingyi Zhou, Xin Wang, Yuchi Ma, Ignacio Iacobacci, Yasheng Wang, Guangtai Liang, Jiansheng Wei, Xin Jiang, Qianxiang Wang and Qun Liu},
  year = {2022},
  doi = {10.48550/arxiv.2207.11280},
  eprint = {2207.11280},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2207.11280}
}
```

- [← previous ID10M: Idiom Identification in 10 Languages](https://md.research.elamur.ai/free/en/papers/tedeschi-2022-id10m-idiom-identification-10-languages/)
- [next → Uncertainty Estimation and Reduction of Pre-trained Models for Text Regression](https://md.research.elamur.ai/free/en/papers/wang-2022-uncertainty-estimation-reduction-pre-trained/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

73 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. NLPaSE 2021 Organization. In 2021 28th Asia-Pacific Software Engineering Conference Workshops (APSEC Workshops), pages 13–13, 2021. doi: 10.1109/APSECW53869.2021.00008. — [doi:10.1109/apsecw53869.2021.00008](https://doi.org/10.1109/apsecw53869.2021.00008) [find](https://scholar.google.com/scholar?q=NLPaSE+2021+Organization%2E+In+2021+28th+Asia%2DPacific+Software+Engineering+Conference+Workshops+%28APSEC+Workshops%29%2C+pages+13%E2%80%9313%2C+2021%2E+doi%3A+10%2E1109%2FAPSECW53869%2E2021%2E00008%2E+%E2%80%94+%5Bdoi%3A10%2E1109%2Fapsecw53869%2E2021)
2. Ehsan Aghaei, Xi Niu, Waseem Shadid, and Ehab Al-Shaer. Language model for text analytic in cybersecurity, 2022. URL https://arxiv.org/abs/2204.02685. — [arXiv:2204.02685](https://arxiv.org/abs/2204.02685) [↗](https://arxiv.org/abs/2204.02685)
3. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. [Unified pre-training for program understanding and generation](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2668, Online,…
4. Uri Alon, Meital Zilberstein, Omer Levy, and Eran Yahav. Code2vec: Learning distributed representations of code. 3(POPL), jan 2019. doi: 10.1145/3290353. URL https://doi.org/10.1145/3290353. [↗](https://doi.org/10.1145/3290353)
5. Sajid Anwar, Mehrdad Saadatmand, Abdul Rauf, Muhammad Ramzan, and Imran Razzak. 2nd International Workshop on Software Engineering Automation: A Natural Language Perspective (NLP-SEA 2021) at ASE 2021. [http://nlpsea.bsoft.pk/](http://nlpsea.bsoft.pk/), 2022. [Online; accessed 9-June-2022]. [↗](http://nlpsea.bsoft.pk/)
6. Dogu Araci. Finbert: Financial sentiment analysis with pre-trained language models. CoRR, abs/1908.10063, 2019. URL http://arxiv.org/abs/1908.10063. [↗](http://arxiv.org/abs/1908.10063)
7. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732, 2021.
8. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Jianfeng Gao, Songhao Piao, Ming Zhou, and Hsiao-Wuen Hon. [UniLMv2: Pseudo-masked language models for unified language model pre-training](https://md.research.elamur.ai/free/en/papers/bao-2020-unilmv2-pseudo-masked-language-models/). In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Con…
9. Iz Beltagy, Arman Cohan, and Kyle Lo. Scibert: Pretrained contextualized embeddings for scientific text. CoRR, abs/1903.[10676, 2019](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/). URL http://arxiv.org/abs/1903.10676. [↗](http://arxiv.org/abs/1903.10676)
10. Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. URL https://doi.org/10.5281/zenodo.5297715. If you use this software, please cite it using these metadata. — [doi:10.5281/zenodo.5297715](https://doi.org/10.5281/zenodo.5297715) [↗](https://doi.org/10.5281/zenodo.5297715)
11. Nadav Brandes, Dan Ofer, Yam Peleg, Nadav Rappoport, and Michal Linial. ProteinBERT: A universal deep-learning model of protein sequence and function. Bioinformatics, 38(8): 2102–2110, 2022. [find](https://scholar.google.com/scholar?q=Nadav+Brandes%2C+Dan+Ofer%2C+Yam+Peleg%2C+Nadav+Rappoport%2C+and+Michal+Linial%2E+ProteinBERT%3A+A+universal+deep%2Dlearning+model+of+protein+sequence+and+function%2E+Bioinformatics%2C+38%288%29%3A+2102%E2%80%932110%2C+2022%2E)
12. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in Neural Information Processing Systems, 33:1877–1901, 2020.
13. Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. [End-to-end object detection with transformers](https://md.research.elamur.ai/free/en/papers/carion-2020-end-end-object-detection-transformers/). In European Conference on Computer Vision, pages 213–229. Springer, 2020.
14. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. [LEGAL-BERT: The muppets straight out of law school](https://md.research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898–2904, Online, November 2020. Association for Computational L…
15. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, p… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Alec+Radford%2C+Rewon+Child%2C+Jeffrey+Wu%2C+Heewoo+Jun%2C+David+Luan%2C+and+Ilya+Sutskever%2E+Generative+pretraining+from+pixels%2E+In+Hal+Daum%C3%A9+III+and+Aarti+Singh%2C+editors%2C+Proceedings+of+the+37th+Int)
16. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
17. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. [A simple framework for contrastive learning of visual representations](https://md.research.elamur.ai/free/en/papers/chen-2020-simple-framework-contrastive-learning-visual/). In Proceedings of the 37th International Conference on Machine Learning, ICML’20. JMLR.org, 2020.
18. Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference on Learning Representations, 2022. URL [https://openreview.net/forum?id=LtKcMgGOeLt](https://openreview.net/forum?id=LtKcMgGOeLt). [↗](https://openreview.net/forum?id=LtKcMgGOeLt)
19. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311, 2022.
20. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020. URL [https://openreview.net/forum?id=r1xMH1BtvB](https://openreview.net/forum?id=r1xMH1BtvB). [↗](https://openreview.net/forum?id=r1xMH1BtvB)
21. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020. URL [https://openreview.net/pdf?id=r1xMH1BtvB](https://openreview.net/pdf?id=r1xMH1BtvB). [↗](https://openreview.net/pdf?id=r1xMH1BtvB)
22. Colin B Clement, Dawn Drain, Jonathan Timcheck, Alexey Svyatkovskiy, and Neel Sundaresan. [PyMT5: Multi-mode translation of natural language and python code with transformers](https://md.research.elamur.ai/free/en/papers/clement-2020-pymt5-multi-mode-translation-natural/). arXiv preprint arXiv:2010.03150, 2020.
23. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
24. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. [Unified Language Model Pre-Training for Natural Language Understanding and Generation](https://md.research.elamur.ai/free/en/papers/dong-2019-unified-language-model-pre-training/). Curran Associates Inc., Red Hook, NY, USA, 2019.
25. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. [An image is worth 16x16 words: Transformers for image recognition at scale](https://md.research.elamur.ai/free/en/papers/dosovitskiy-2020-image-worth-16x16-words-transformers/). In Internationa…
26. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. [CodeBERT: A pre-trained model for programming and natural languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547, Online…
27. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999, 2022. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthesis%2E)
28. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling. CoRR, abs/2101.00027, 2021. URL https://arxiv.org/abs/2101.00027… [↗](https://arxiv.org/abs/2101.00027…)
29. Georgios Gousios. The ghtorrent dataset and tool suite. In Proceedings of the 10th Working Conference on Mining Software Repositories, MSR ’13, pages 233–236, Piscataway, NJ, USA, 2013. IEEE Press. ISBN 978-1-4673-2936-1. URL [http://dl.acm.org/citation.cfm?id=2487085.2487132](http://dl.acm.org/citation.cfm?id=2487085.2487132). [↗](http://dl.acm.org/citation.cfm?id=2487085.2487132)
30. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie LIU, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. Graphcode{bert}: Pre-training code representations with data flow… [find](https://scholar.google.com/scholar?q=Daya+Guo%2C+Shuo+Ren%2C+Shuai+Lu%2C+Zhangyin+Feng%2C+Duyu+Tang%2C+Shujie+LIU%2C+Long+Zhou%2C+Nan+Duan%2C+Alexey+Svyatkovskiy%2C+Shengyu+Fu%2C+Michele+Tufano%2C+Shao+Kun+Deng%2C+Colin+Clement%2C+Dawn+Drain%2C+Neel+Sundaresan%2C+Jia)
31. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. [UniXcoder: Unified cross-modal pre-training for code representation](https://md.research.elamur.ai/free/en/papers/guo-2022-unixcoder-unified-cross-modal-pre/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7212–7225, Dublin, Ireland, May 2022. Ass…
32. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, Qin Jin, Yanyan Lan, Yang Liu, Zhiyuan Liu, Zhiwu Lu, Xipeng Qiu, Ruihua Song, Jie Tang, Ji-Rong Wen, Jinhui Yuan, Wayne Xin Zhao, and Jun Zhu. Pre-trained models: Past, present and… [find](https://scholar.google.com/scholar?q=Xu+Han%2C+Zhengyan+Zhang%2C+Ning+Ding%2C+Yuxian+Gu%2C+Xiao+Liu%2C+Yuqi+Huo%2C+Jiezhong+Qiu%2C+Liang+Zhang%2C+Wentao+Han%2C+Minlie+Huang%2C+Qin+Jin%2C+Yanyan+Lan%2C+Yang+Liu%2C+Zhiyuan+Liu%2C+Zhiwu+Lu%2C+Xipeng+Qiu%2C+Ruihua+Song%2C+Ji)
33. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. [Momentum contrast for unsupervised visual representation learning](https://md.research.elamur.ai/free/en/papers/he-2019-momentum-contrast-unsupervised-visual-representation/). In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9726–9735, 2020. doi: 10.1109/CVPR42600.2020.00975.
34. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Sys…
35. Abram Hindle, Earl T. Barr, Zhendong Su, Mark Gabel, and Premkumar Devanbu. On the naturalness of software. ICSE ’12, page 837–847. IEEE Press, 2012. ISBN 9781467310673. [find](https://scholar.google.com/scholar?q=Abram+Hindle%2C+Earl+T%2E+Barr%2C+Zhendong+Su%2C+Mark+Gabel%2C+and+Premkumar+Devanbu%2E+On+the+naturalness+of+software%2E+ICSE+%E2%80%9912%2C+page+837%E2%80%93847%2E+IEEE+Press%2C+2012%2E+ISBN+9781467310673%2E)
36. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. [The curious case of neural text degeneration](https://md.research.elamur.ai/free/en/papers/holtzman-2019-curious-case-neural-text-degeneration/). In ICLR, 2020. URL [http://dblp.uni-trier.de/db/conf/iclr/iclr2020.html\#HoltzmanBDFC20](http://dblp.uni-trier.de/db/conf/iclr/iclr2020.html\#HoltzmanBDFC20). [↗](http://dblp.uni-trier.de/db/conf/iclr/iclr2020.html\#HoltzmanBDFC20)
37. Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. [Clinicalbert: Modeling clinical notes and predicting hospital readmission](https://md.research.elamur.ai/free/en/papers/huang-2019-clinicalbert-modeling-clinical-notes-predicting/). CoRR, abs/1904.05342, 2019. URL http://arxiv.org/abs/1904.05342. [↗](http://arxiv.org/abs/1904.05342)
38. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. [Codesearchnet challenge: Evaluating the state of semantic code search](https://md.research.elamur.ai/free/en/papers/husain-2020-codesearchnet-challenge-evaluating-state-semantic/). CoRR, abs/1909.09436, 2019. URL http://arxiv.org/abs/1909.09436. [↗](http://arxiv.org/abs/1909.09436)
39. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643–1652, Brussels, Belgium, October-November 2018. Association for Computational… [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Ioannis+Konstas%2C+Alvin+Cheung%2C+and+Luke+Zettlemoyer%2E+Mapping+language+to+code+in+programmatic+context%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+P)
40. Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. Contrastive code representation learning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5954–5971, Online and Punta Cana, Dominican Republic, November 2021. Associ… [find](https://scholar.google.com/scholar?q=Paras+Jain%2C+Ajay+Jain%2C+Tianjun+Zhang%2C+Pieter+Abbeel%2C+Joseph+Gonzalez%2C+and+Ion+Stoica%2E+Contrastive+code+representation+learning%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+La)
41. Xue Jiang, Zhuoran Zheng, Chen Lyu, Liang Li, and Lei Lyu. Treebert: A tree-based pre-trained model for programming language. In Uncertainty in Artificial Intelligence, pages 54–63. PMLR, 2021. [find](https://scholar.google.com/scholar?q=Xue+Jiang%2C+Zhuoran+Zheng%2C+Chen+Lyu%2C+Liang+Li%2C+and+Lei+Lyu%2E+Treebert%3A+A+tree%2Dbased+pre%2Dtrained+model+for+programming+language%2E+In+Uncertainty+in+Artificial+Intelligence%2C+pages+54%E2%80%9363%2E+PMLR%2C+2021%2E)
42. Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. Pre-trained contextual embedding of source code, 2020. URL [https://openreview.net/forum?id=rygoURNYvS](https://openreview.net/forum?id=rygoURNYvS). [↗](https://openreview.net/forum?id=rygoURNYvS)
43. Diederik P. Kingma and Jimmy Ba. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http://arxiv.org/abs/1412.6980. [↗](http://arxiv.org/abs/1412.6980)
44. Taku Kudo and John Richardson. [SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66– 71, Brussels, Belgium, November 20…
45. Royi Lachmy, Ziyu Yao, Greg Durrett, Milos Gligoric, Junyi Jessy Li, Ray Mooney, Graham Neubig, Yu Su, Huan Sun, and Reut Tsarfaty, editors. Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), Online, August 2021. Association for Computational Linguistics.… — [ACL 2021.nlp4prog-1.0](https://aclanthology.org/2021.nlp4prog-1.0/) [find](https://scholar.google.com/scholar?q=Royi+Lachmy%2C+Ziyu+Yao%2C+Greg+Durrett%2C+Milos+Gligoric%2C+Junyi+Jessy+Li%2C+Ray+Mooney%2C+Graham+Neubig%2C+Yu+Su%2C+Huan+Sun%2C+and+Reut+Tsarfaty%2C+editors%2E+Proceedings+of+the+1st+Workshop+on+Natural+Language+Process)
46. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. [Biobert: a pre-trained biomedical language representation model for biomedical text mining](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/). Bioinformatics, 36(4):1234–1240, 2020.
47. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
48. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. Codexglue: A… [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+B%2E+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+Ge+Li%2C+Lidong+Zhou%2C+Linjun+Shou%2C+Long+Zhou%2C+Michele+Tufano%2C+Ming)
49. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. [Efficient estimation of word representations in vector space](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/). arXiv preprint arXiv:1301.3781, 2013.
50. Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madelei… — [Text and Code Embeddings by Contrastive Pre-Training](https://md.research.elamur.ai/free/en/papers/neelakantan-2022-text-code-embeddings-contrastive-pre/)
51. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13474, 2022. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+A+conversational+paradigm+for+program+synthesis%2E+arXiv+preprint+arXiv%3A2203%2E13474%2C+2022%2E)
52. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computat…
53. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. [Pytorch: An imperative style, high-performance deep learning library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/). Advances in neural information processing systems, 32, 2019.
54. Long Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James Annibal, Alec Peltekian, and Yanfang Ye. CoTexT: Multi-task learning with code-text transformer. In Proceedings of the 1st Workshop on Natural Language Processing for Programming (NLP4Prog 2021), pages 40–47, Online, August 2021. Association for Co… [find](https://scholar.google.com/scholar?q=Long+Phan%2C+Hieu+Tran%2C+Daniel+Le%2C+Hieu+Nguyen%2C+James+Annibal%2C+Alec+Peltekian%2C+and+Yanfang+Ye%2E+CoTexT%3A+Multi%2Dtask+learning+with+code%2Dtext+transformer%2E+In+Proceedings+of+the+1st+Workshop+on+Natural+Langu)
55. Jiahao Qin and Lu Zong. TS-BERT: A fusion model for pre-trainning time series-text representations, 2022. URL [https://openreview.net/forum?id=Fia60I79-4B](https://openreview.net/forum?id=Fia60I79-4B). [↗](https://openreview.net/forum?id=Fia60I79-4B)
56. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+2018%2E)
57. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+Language+models+are+unsupervised+multitask+learners%2E+2019%2E)
58. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67, 2020. URL [http://jmlr.org/papers/v21/20…](http://jmlr.org/papers/v21/20…) [↗](http://jmlr.org/papers/v21/20…)
59. Veselin Raychev, Martin Vechev, and Eran Yahav. Code completion with statistical language models. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 419–428, 2014. [find](https://scholar.google.com/scholar?q=Veselin+Raychev%2C+Martin+Vechev%2C+and+Eran+Yahav%2E+Code+completion+with+statistical+language+models%2E+In+Proceedings+of+the+35th+ACM+SIGPLAN+Conference+on+Programming+Language+Design+and+Implementation%2C+p)
60. Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. Codebleu: a method for automatic evaluation of code synthesis. CoRR, abs/2009.10297, 2020. URL https://arxiv.org/abs/2009.10297. [↗](https://arxiv.org/abs/2009.10297)
61. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. [Unsupervised translation of programming languages](https://md.research.elamur.ai/free/en/papers/lachaux-2020-unsupervised-translation-programming-languages/). Advances in Neural Information Processing Systems, 33:20601–20611, 2020.
62. Torsten Scholak, Gabriel Orlanski, Disha Shrivastava, Arun Raja, Dzmitry Bahdanau, and Jonathan Herzig. Deep Learning For Code (DL4C) Workshop at ICLR 2022. [https://dl4c.github.io/](https://dl4c.github.io/), 2022. [Online; accessed 9-June-2022]. [↗](https://dl4c.github.io/)
63. Andrea Di Sorbo, Sebastiano Panichella, Oscar Chaparro, Rafael Kallis, and Yang Song. The 1st Intl. Workshop on Natural Language-based Software Engineering Co-located with ICSE 2022. [https://nlbse2022.github.io/](https://nlbse2022.github.io/), 2022. [Online; accessed 20-June-2022]. [↗](https://nlbse2022.github.io/)
64. Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. [Training data-efficient image transformers & distillation through attention](https://md.research.elamur.ai/free/en/papers/touvron-2020-training-data-efficient-image-transformers/). In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 13…
65. Trieu H. Trinh, Minh-Thang Luong, and Quoc V. Le. Selfie: Self-supervised pretraining for image embedding. CoRR, abs/1906.02940, 2019. URL http://arxiv.org/abs/1906.02940. [↗](http://arxiv.org/abs/1906.02940)
66. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in Neural Information Processing Systems, 30, 2017.
67. Ben Wang. Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax), May 2021. [↗](https://github.com/kingoflolz/mesh-transformer-jax)
68. Xin Wang, Yasheng Wang, Pingyi Zhou, Fei Mi, Meng Xiao, Yadao Wang, Li Li, Xiao Liu, Hao Wu, Jin Liu, and Xin Jiang. CLSEBERT: contrastive learning for syntax enhanced code pre-trained model. CoRR, abs/2108.04556, 2021. URL https://arxiv.org/abs/2108.04556. [↗](https://arxiv.org/abs/2108.04556)
69. Xin Wang, Yasheng Wang, Yao Wan, Jiawei Wang, Pingyi Zhou, Li Li, Hao Wu, and Jin Liu. Code-mvp: Learning to represent source code from multiple views with contrastive pre-training, 2022. URL https://arxiv.org/abs/2205.02029. [↗](https://arxiv.org/abs/2205.02029)
70. Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696–8708, Online and Punta Cana, Domin… [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+C%2EH%2E+Hoi%2E+CodeT5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+In+Proceedings+of+the+2021+Conferenc)
71. Neo Wu, Bradley Green, Xue Ben, and Shawn O’Banion. Deep transformer models for time series forecasting: The influenza prevalence case. CoRR, abs/2001.08317, 2020. URL https://arxiv.org/abs/2001.08317. — [arXiv:2001.08317](https://arxiv.org/abs/2001.08317) [↗](https://arxiv.org/abs/2001.08317)
72. Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’20, page 753–763, New Yo… — [doi:10.1145/3394486.3403118](https://doi.org/10.1145/3394486.3403118) [find](https://scholar.google.com/scholar?q=Zonghan+Wu%2C+Shirui+Pan%2C+Guodong+Long%2C+Jing+Jiang%2C+Xiaojun+Chang%2C+and+Chengqi+Zhang%2E+Connecting+the+dots%3A+Multivariate+time+series+forecasting+with+graph+neural+networks%2E+In+Proceedings+of+the+26th+ACM)
73. Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, Chen Li, Ziyan Gong, Yifan Yao, Xinjing Huang, Jun Wang, Jianfeng Yu, Qi Guo, Yue Yu, Yan Zhang, Jin Wang, Hengtao Tao, Dasen Yan, Zexuan Yi, Fang Peng, Fangqing Jiang, Han Zhang,… — [PanGu-$\alpha$: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation](https://md.research.elamur.ai/free/en/papers/zeng-2021-pangu-alpha-large-scale/)

*Updated 31 October 2026*