--- title: "UniXcoder: Unified Cross-Modal Pre-training for — Guo 2022" description: "Pre-trained code models usually fall into one of three types, and each has a weakness." authors: ["Daya Guo", "Shuai Lu", "Nan Duan", "Yanlin Wang", "Ming Zhou", "Jian Yin"] year: 2022 venue: "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)" source: "https://aclanthology.org/2022.acl-long.499/" doi: "10.18653/v1/2022.acl-long.499" arxiv: "2203.03850" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/guo-2022-unixcoder-unified-cross-modal-pre/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/guo-2022-unixcoder-unified-cross-modal-pre/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) / UniXcoder: Unified Cross-Modal Pre-training for Code Representation # UniXcoder: Unified Cross-Modal Pre-training for Code Representation **Authors:** [Daya Guo](https://md.research.elamur.ai/free/en/authors/daya-guo/), [Shuai Lu](https://md.research.elamur.ai/free/en/authors/shuai-lu/), [Nan Duan](https://md.research.elamur.ai/free/en/authors/nan-duan/), [Yanlin Wang](https://md.research.elamur.ai/free/en/authors/yanlin-wang/), [Ming Zhou](https://md.research.elamur.ai/free/en/authors/ming-zhou/), [Jian Yin](https://md.research.elamur.ai/free/en/authors/jian-yin/) · **Year:** 2022 · **Venue:** Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.acl-long.499/) · [PDF](https://aclanthology.org/2022.acl-long.499.pdf) · [DOI 10.18653/v1/2022.acl-long.499](https://doi.org/10.18653/v1/2022.acl-long.499) · [arXiv 2203.03850](https://arxiv.org/abs/2203.03850) · [OpenAlex](https://openalex.org/W4221166942) **Citations:** 636 citations (openalex-snapshot · 2026-10-14) · [132](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Pre-trained code models usually fall into one of three types, and each has a weakness. Encoder-only models need a decoder trained from scratch to generate text or code. Decoder-only models are weaker at understanding tasks. Encoder-decoder models are inefficient for code completion, which in practice should run decoder-only. UniXcoder addresses this with one multi-layer Transformer. Mask attention matrices combined with prefix adapters switch the model between encoder, decoder and encoder-decoder modes. Besides source code, the model takes code comments and the AST as input. The AST is flattened by a reversible (one-to-one) mapping that keeps the full tree structure. Pre-training mixes three language-modeling objectives with two representation tasks: multi-modal contrastive learning and cross-modal generation, which uses comments to align embeddings across programming languages. The authors evaluated UniXcoder on five task types over nine datasets, plus a new zero-shot code-to-code search benchmark built from CodeNet. It reached state-of-the-art results on most tasks, and ablations show that both comments and AST help. This makes UniXcoder a practical single backbone for both understanding and generation in code intelligence. ## Key points - Pre-training objectives are masked language modeling, unidirectional language modeling and a denoising objective. Two further tasks target code-fragment embeddings: contrastive learning and cross-modal generation. - The AST mapping wraps each non-terminal node's children with left and right boundary tokens. Swapping it for BFS or DFS traversal lowered scores, sometimes below the variant with no AST at all. - The new zero-shot code-to-code search dataset contains 11,744 Ruby, 15,594 Python and 23,530 Java functions from CodeNet, covering 4,053 problems. It is scored with MAP. - On this benchmark UniXcoder reaches an overall MAP of 20.45%, about 11 points above GraphCodeBERT. Without contrastive learning the score drops to 13.73%. - For code completion, accuracy on the Java corpus rose by an absolute 2.3%, and UniXcoder outperformed PLBART and CodeT5. Code and data are released at github.com/microsoft/CodeBERT. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/) ## Cited by Cited by 132 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 125 not yet. - [基于大语言模型智能体的代码生成综述](https://md.research.elamur.ai/free/en/papers/dong-2025-code-generation-llm/) - [预训练模型在软件工程领域应用研究进展](https://md.research.elamur.ai/free/en/papers/gong-2024-pre-trained-model/) - [A Survey on Large Language Models for Code Generation](https://md.research.elamur.ai/free/en/papers/jiang-2024-survey-large-language-models-code/) - [xCodeEval: A Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://md.research.elamur.ai/free/en/papers/khan-2023-xcodeeval-large-scale-multilingual-multitask/) - [XCodeEval: An Execution-based Large Scale Multilingual Multitask Benchmark for Code Understanding, Generation, Translation and Retrieval](https://md.research.elamur.ai/free/en/papers/khan-2024-xcodeeval-execution-based-large-scale/) - [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://md.research.elamur.ai/free/en/papers/li-2023-towards-general-text-embeddings-multi/) - [基于深度学习的代码生成方法研究进展](https://md.research.elamur.ai/free/en/papers/yang-2023-deep-learning-code/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{guo2022unixcoder, title = {{UniXcoder: Unified Cross-Modal Pre-training for Code Representation}}, author = {Guo, Daya and Lu, Shuai and Duan, Nan and Wang, Yanlin and Zhou, Ming and Yin, Jian}, booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)}, year = {2022}, doi = {10.18653/v1/2022.acl-long.499}, eprint = {2203.03850}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/2022.acl-long.499} } ``` - [← previous Universal Grammatical Dependencies for Portuguese with CINTIL Data, LX Processing and CLARIN support](https://md.research.elamur.ai/free/en/papers/branco-2022-universal-grammatical-dependencies-portuguese-cintil/) - [next → Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval](https://md.research.elamur.ai/free/en/papers/gao-2022-unsupervised-corpus-aware-language-model/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/5/) ## References 5 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2021. Unified pre-training for program understanding and generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2668. [find](https://scholar.google.com/scholar?q=Wasi+Ahmad%2C+Saikat+Chakraborty%2C+Baishakhi+Ray%2C+and+Kai%2DWei+Chang%2E+2021%2E+Unified+pre%2Dtraining+for+program+understanding+and+generation%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapt) [our card](https://md.research.elamur.ai/free/en/papers/ahmad-2021-unified-pre-training-program-understanding/) 2. Miltiadis Allamanis and Charles Sutton. 2013. Mining source code repositories at massive scale using language modeling. In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 207– 216. IEEE. [find](https://scholar.google.com/scholar?q=Miltiadis+Allamanis+and+Charles+Sutton%2E+2013%2E+Mining+source+code+repositories+at+massive+scale+using+language+modeling%2E+In+2013+10th+Working+Conference+on+Mining+Software+Repositories+%28MSR%29%2C+pages+207) 3. GraphCodeBERT: def ranks(self, key, value): return [normalize_rank(el) for el in force_list(value.get('a'))] PLBART: def zrank(self, name, value): with self.pipe as pipe: value = self.valueparse.encode(value) return pipe.zrank(self.redis_key(name), value) [find](https://scholar.google.com/scholar?q=GraphCodeBERT%3A+def+ranks%28self%2C+key%2C+value%29%3A+return+%5Bnormalize%5Frank%28el%29+for+el+in+force%5Flist%28value%2Eget%28%27a%27%29%29%5D+PLBART%3A+def+zrank%28self%2C+name%2C+value%29%3A+with+self%2Epipe+as+pipe%3A+value+%3D+self%2Evalueparse%2Eencod) 4. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165. [find](https://scholar.google.com/scholar?q=Tom+B+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+et+al%2E+2020%2E+Language+models+are+few%2Dshot+lear) 5. CodeT5: def sort_key(x): name, (r, u) = x return - len(u) + u.count('}') * 100 UniXcoder: def revrank_dict(dict, key=lambda t: t[1], as_tuple=False): sorted_list = sorted(dict.items(), key=key, reverse=True) return OrderedDict(sorted_list) if not as_tuple else tuple(sorted_list) *Figure 4: An exampl… [find](https://scholar.google.com/scholar?q=CodeT5%3A+def+sort%5Fkey%28x%29%3A+name%2C+%28r%2C+u%29+%3D+x+return+%2D+len%28u%29+%2B+u%2Ecount%28%27%7D%27%29+%2A+100+UniXcoder%3A+def+revrank%5Fdict%28dict%2C+key%3Dlambda+t%3A+t%5B1%5D%2C+as%5Ftuple%3DFalse%29%3A+sorted%5Flist+%3D+sorted%28dict%2Eitems%28%29%2C+key%3Dkey%2C+revers)