--- title: "CPT — paper by Shao et al. (2021) — summary and ideas" description: "The authors propose CPT, which splits a Transformer into three parts. 7 references, cited by 67 papers here." authors: ["Yunfan Shao", "Zhichao Geng", "Yitao Liu", "Junqi Dai", "Hang Yan", "Fei Yang", "Li Zhe", "Hujun Bao", "Xipeng Qiu"] year: 2021 venue: "arXiv:2109.05729" source: "https://arxiv.org/abs/2109.05729" doi: "10.1007/s11432-021-3536-5" arxiv: "2109.05729" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/shao-2021-cpt-pre-trained-unbalanced-transformer/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shao-2021-cpt-pre-trained-unbalanced-transformer/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation # CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation **Authors:** [Yunfan Shao](https://md.research.elamur.ai/free/en/authors/yunfan-shao/), Zhichao Geng, [Yitao Liu](https://md.research.elamur.ai/free/en/authors/yitao-liu/), [Junqi Dai](https://md.research.elamur.ai/free/en/authors/junqi-dai/), [Hang Yan](https://md.research.elamur.ai/free/en/authors/hang-yan/), [Fei Yang](https://md.research.elamur.ai/free/en/authors/fei-yang/), Li Zhe, Hujun Bao and [Xipeng Qiu](https://md.research.elamur.ai/free/en/authors/xipeng-qiu/) · **Year:** 2021 · **Venue:** [arXiv:2109.05729](https://md.research.elamur.ai/free/en/venues/science-china-information-sciences/) ([Science China Information Sciences 2021](https://md.research.elamur.ai/free/en/venues/science-china-information-sciences/2021/)) · **Publisher:** [Springer Nature](https://md.research.elamur.ai/free/en/publishers/springer-nature/) · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://doi.org/10.1007/s11432-021-3536-5) · [PDF](https://arxiv.org/pdf/2109.05729) · [DOI 10.1007/s11432-021-3536-5](https://doi.org/10.1007/s11432-021-3536-5) · [arXiv 2109.05729](https://arxiv.org/abs/2109.05729) · [OpenAlex](https://openalex.org/W3199824684) **Citations:** 113 citations (OpenAlex, 2026-10-14) · [67](https://md.research.elamur.ai/free/en/papers/shao-2021-cpt-pre-trained-unbalanced-transformer/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Chinese pre-trained models are usually built either for understanding (BERT-style encoders) or for generation (GPT- or BART-style models). This forces practitioners to choose a backbone per task and prevents knowledge from being shared between NLU and NLG. The authors propose CPT, which splits a Transformer into three parts. A deep shared encoder (S-Enc) feeds two shallow heads. The first is an understanding decoder (U-Dec) with full self-attention, trained with masked language modeling using Whole Word Masking. The second is a generation decoder (G-Dec) with causal attention and cross-attention, trained with BART-style denoising auto-encoding. Both objectives are optimized jointly on unlabeled text. The two branches can be combined in several ways at fine-tuning time. The deep-shallow layout keeps the model's size close to BERT and BART, and the light generation decoder makes decoding much faster. Experiments covered classification, sequence labeling, reading comprehension, summarization and data-to-text tasks. CPT stayed competitive with strong Chinese baselines while generating faster than a comparable Chinese BART. This makes it a practical single backbone for mixed Chinese NLP workloads. ## Key points - The architecture is unbalanced: a deep S-Enc plus two shallow decoders. U-Dec is pre-trained with MLM using WWM, and G-Dec with DAE using token infilling (one [MASK] per selected word) and sentence permutation. - Masking selects 15% of words after Chinese Word Segmentation. Of the masked characters, BERT's scheme replaces 80% with [MASK] and 10% with random tokens. - Classification tasks can be fine-tuned in at least five modes, using different combinations of the encoder and decoders. - Generation runs about 1.4–1.5× faster than BART at base size and up to 1.7× faster at large size, with comparable output quality. Evaluation sets include ADGEN data-to-text. - A Chinese BART is released as a by-product, and the code is available at github.com/fastnlp/CPT. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Fudan University](https://md.research.elamur.ai/free/en/universities/fudan-university/) — University, Shanghai, China · 202 works · 22,509 citations - [Zhejiang Lab](https://md.research.elamur.ai/free/en/labs/zhejiang-lab/) — Research institute, Hangzhou, China · 9 works · 1,776 citations 📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works ## Cite Cite as: Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao and Xipeng Qiu. 2021. CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation. arXiv:2109.05729. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{shao2021cpt, title = {{CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation}}, author = {Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao and Xipeng Qiu}, year = {2021}, doi = {10.1007/s11432-021-3536-5}, eprint = {2109.05729}, archivePrefix = {arXiv}, url = {https://doi.org/10.1007/s11432-021-3536-5} } ``` - [← previous MediaSum: A Large-scale Media Interview Dataset for Dialogue Summarization](https://md.research.elamur.ai/free/en/papers/zhu-2021-mediasum-large-scale-media-interview/) - [next → Glancing Transformer for Non-Autoregressive Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/qian-2021-glancing-transformer-non-autoregressive-neural/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) ## References 7 references, 5 of them with a card of ours. ↗ leads to the work. 1. X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” SCIENCE CHINA Technological Sciences, vol. 63, no. 10, p. 1872[–1897, 2020](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for l… 2. PMLR, 2020, pp. 642–652. [Online]. Available: [http://proceedings.mlr.press/v119/bao20a.html](http://proceedings.mlr.press/v119/bao20a.html) Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “All NLP tasks are generation tasks: A general pretraining framework,” CoRR, vol. abs/2103.10360, 2021. [Online]. Available: https://arxiv.org/a… [↗](http://proceedings.mlr.press/v119/bao20a.html) 3. AAAI Press, 2020, pp. 8968–8975. [Online]. Available: — [GSum: A General Framework for Guided Neural Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/dou-2021-gsum-general-framework-guided-neural/) [https://aaai.org/ojs/index.php/AAAI/article/view/6428](https://aaai.org/ojs/index.php/AAAI/article/view/6428) Z. Sun, X. Li, X. Sun, Y. Meng, X. Ao, Q. He, F. Wu, and J. Li, “Chinesebert: Chinese pretraining enhanced by glyph and pinyin information,” CoRR, vol. abs/2106.16038, 2021. [Online]. Availabl… [↗](https://aaai.org/ojs/index.php/AAAI/article/view/6428) 4. Z. Lin, A. Madotto, G. I. Winata, and P. Fung, “[Mintl: Minimalist transfer learning for task-oriented dialogue systems](https://md.research.elamur.ai/free/en/papers/lin-2020-mintl-minimalist-transfer-learning-task/),” in EMNLP 2020, Online, November 16-20, 2020, B. Webber, T. Cohn, Y. He, and Y. Liu, Eds., 2020, pp. 3391–3405. [Online]. Available: https://doi.org/10.18653/v1/2020.emnlp-main.273… [↗](https://doi.org/10.18653/v1/2020.emnlp-main.273…) 5. OpenReview.net, 2021. [Online]. Available: [https://openreview.net/forum?id=KpfasTaLUpq](https://openreview.net/forum?id=KpfasTaLUpq) X. Sun, T. Ge, F. Wei, and H. Wang, “Instantaneous grammatical error correction with shallow aggressive decoding,” in ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, C. Zong, F. Xia, W.… [↗](https://openreview.net/forum?id=KpfasTaLUpq) 6. ACL, 2005. [Online](https://md.research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/). Available: [https://www.aclweb.org/anthology/I05-3017/G](https://www.aclweb.org/anthology/I05-3017/G). Levow, “The third international chinese language processing bakeoff: Word segmentation and named entity recognition,” in SIGHAN@COLING/ACL 2006, Sydney, Australia, July 22-23, 2006, H. T. Ng and O. O. Y. Kwong, Eds., 2006,… [↗](https://www.aclweb.org/anthology/I05-3017/G) 7. EMNLP 2020, 2020, pp. 2887–2897. Y. Cui, T. Liu, W. Che, L. Xiao, Z. Chen, W. Ma, S. Wang, and G. Hu, “[A span-extraction dataset for chinese machine reading comprehension](https://md.research.elamur.ai/free/en/papers/cui-2018-span-extraction-dataset-chinese/),” in EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., 2019, pp. 5882–5888. [O… *Updated 2 November 2026*