---
title: "Sub-Character Tokenization for Chinese Pretrained Language"
description: "They propose sub-character (SubChar) tokenization. 59 references, cited by 14 papers here."
authors: ["Chenglei Si", "Zhengyan Zhang", "Yingfa Chen", "Fanchao Qi", "Xiaozhi Wang", "Zhiyuan Liu", "Yasheng Wang", "Qun Liu", "Maosong Sun"]
year: 2023
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2023.tacl-1.28/"
doi: "10.1162/tacl_a_00560"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/si-2023-sub-character-tokenization-chinese-pretrained/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/si-2023-sub-character-tokenization-chinese-pretrained/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/) / Sub-Character Tokenization for Chinese Pretrained Language Models

# Sub-Character Tokenization for Chinese Pretrained Language Models

**Authors:** [Chenglei Si](https://md.research.elamur.ai/free/en/authors/chenglei-si/), [Zhengyan Zhang](https://md.research.elamur.ai/free/en/authors/zhengyan-zhang/), [Yingfa Chen](https://md.research.elamur.ai/free/en/authors/yingfa-chen/), [Fanchao Qi](https://md.research.elamur.ai/free/en/authors/fanchao-qi/), [Xiaozhi Wang](https://md.research.elamur.ai/free/en/authors/xiaozhi-wang/), [Zhiyuan Liu](https://md.research.elamur.ai/free/en/authors/zhiyuan-liu/), [Yasheng Wang](https://md.research.elamur.ai/free/en/authors/yasheng-wang/), [Qun Liu](https://md.research.elamur.ai/free/en/authors/qun-liu/) and [Maosong Sun](https://md.research.elamur.ai/free/en/authors/maosong-sun/) · **Year:** 2023 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2023](https://md.research.elamur.ai/free/en/venues/tacl/2023/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.tacl-1.28/) · [PDF](https://aclanthology.org/2023.tacl-1.28.pdf) · [DOI 10.1162/tacl\_a\_00560](https://doi.org/10.1162/tacl_a_00560) · [OpenAlex](https://openalex.org/W4377116057)

**Citations:** 15 citations (OpenAlex, 2026-10-14) · [14](https://md.research.elamur.ai/free/en/papers/si-2023-sub-character-tokenization-chinese-pretrained/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Chinese pretrained language models usually tokenize text either one character at a time or by running sub-word algorithms directly on raw characters. The authors argue that both approaches miss information stored inside characters. Glyphs are built from radicals that carry meaning, and pronunciation cannot be read reliably from the script. They propose sub-character (SubChar) tokenization. First, each character is rewritten as a short code string. This can be a phonetic transliteration (pinyin, zhuyin) or a shape-based code (stroke order, or the radicals of the Wubi, Zhengma and Cangjie input methods). Then a standard sub-word segmenter learns a vocabulary over these codes. Chinese word segmentation can optionally be applied first. The authors pretrained BERT-style models with these tokenizers and compared them with character and sub-word baselines on more than ten NLU datasets. Downstream accuracy stays comparable. Tokenized sequences become much shorter, which speeds up pretraining and fine-tuning. Pronunciation-based variants give homophones identical codes, so models resist homophone typos. The work shows that properties of a specific writing system can produce cheaper and more robust models.

## Key points
- Six character encodings were tested: two pronunciation-based (pinyin, zhuyin) and four glyph-based (Stroke, Wubi, Zhengma, Cangjie).
  - A '#' separator marks the end of each character's code.
  - Numeric indices distinguish characters that share a code. The SubChar-Pinyin-NoIndex variant drops these indices to test whether disambiguation matters.
- The default segmenter is the unigram language model in SentencePiece. An ablation with BPE gave similar accuracy and efficiency, so the gains do not depend on the segmenter.
- On the iFLYTEK long-text classification dataset, SubChar tokenizers shortened tokenized output by up to about 40% at the same vocabulary size as CharTokenizer. A few sub-character tokens can compose many rare characters, which leaves vocabulary room for multi-character words and phrases.
- In the optional word segmentation step, THULAC splits text into words, and frequent words form about 80% of the vocabulary. Words outside the vocabulary fall back to SubChar tokenization.
- Code and models are released at github.com/thunlp/SubCharTokenization. The stated limitations are that only NLU tasks were evaluated (not generation) and that the method is designed for Chinese only.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Tsinghua University](https://md.research.elamur.ai/free/en/universities/tsinghua-university/) — University, Beijing, China · 333 works · 87,019 citations
- [University of Maryland, College Park](https://md.research.elamur.ai/free/en/universities/university-of-maryland-college-park/) — University, College Park, United States · 132 works · 25,380 citations
- [Huawei Technologies](https://md.research.elamur.ai/free/en/companies/huawei-technologies/) — Company, Shenzhen, China · 75 works · 12,975 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 119 works

## Cite

Cite as: Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu and Maosong Sun. 2023. Sub-Character Tokenization for Chinese Pretrained Language Models. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{si2023sub,
  title = {{Sub-Character Tokenization for Chinese Pretrained Language Models}},
  author = {Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, Yasheng Wang, Qun Liu and Maosong Sun},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00560},
  url = {https://doi.org/10.1162/tacl_a_00560}
}
```

- [← previous SALMA: Arabic Sense-Annotated Corpus and WSD Benchmarks](https://md.research.elamur.ai/free/en/papers/jarrar-2023-salma-arabic-sense-annotated-corpus/)
- [next → Towards Reasoning in Large Language Models via Multi-Agent Peer Review Collaboration](https://md.research.elamur.ai/free/en/papers/xu-2023-towards-reasoning-large-language-models/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/9/)

## References

59 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emily Bender. 2019. The \#BenderRule: On naming the languages we study and why it matters. The Gradient. [find](https://scholar.google.com/scholar?q=Emily+Bender%2E+2019%2E+The+%5C%23BenderRule%3A+On+naming+the+languages+we+study+and+why+it+matters%2E+The+Gradient%2E)
2. Akash Bharadwaj, David Mortensen, Chris Dyer, and Jaime Carbonell. 2016. Phonologically aware neural model for named entity recognition in low resource transfer settings. In Proceedings of EMNLP, pages 1462–1472. https://doi.org/10.18653/v1/D16 -1153 [↗](https://doi.org/10.18653/v1/D16)
3. Shaosheng Cao, Wei Lu, Jun Zhou, and Xiaolong Li. 2018. cw2vec: Learning chinese word embeddings with stroke n-gram information. In Proceedings of AAAI. [find](https://scholar.google.com/scholar?q=Shaosheng+Cao%2C+Wei+Lu%2C+Jun+Zhou%2C+and+Xiaolong+Li%2E+2018%2E+cw2vec%3A+Learning+chinese+word+embeddings+with+stroke+n%2Dgram+information%2E+In+Proceedings+of+AAAI%2E)
4. Pi-Chuan Chang, Michel Galley, and Christopher D. Manning. 2008. [Optimizing Chinese word segmentation for machine translation performance](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/). In Proceedings of the Third Workshop on Statistical Machine Translation. https://doi.org/10.3115/1626394.1626430 \#\# Acknowledgments This work is supported by th… [↗](https://doi.org/10.3115/1626394.1626430)
5. Aditi Chaudhary, Chunting Zhou, Lori Levin, Graham Neubig, David R. Mortensen, and Jaime Carbonell. 2018. Adapting word embeddings to new languages with morphological and phonological subword representations. In Proceedings of EMNLP, pages 3285–3295. https://doi.org/10.18653/v1/D18 -1366 [↗](https://doi.org/10.18653/v1/D18)
6. Jing Chen, Qingcai Chen, Xin Liu, Haijun Yang, Daohe Lu, and Buzhou Tang. 2018. The BQ corpus: A large-scale domain-specific Chinese corpus for sentence semantic equivalence identification. In Proceedings of EMNLP. https://doi.org/10.18653/v1/D18 -1536 cessing of Oriental Languages. https://doi.org… [↗](https://doi.org/10.18653/v1/D18)
7. Yen-Chen Hao and Chung-Lin Martin Yang. 2021. The effect of second-language orthographic input on the phonological encoding of Mandarin words. Applied Psycholinguistics. [find](https://scholar.google.com/scholar?q=Yen%2DChen+Hao+and+Chung%2DLin+Martin+Yang%2E+2021%2E+The+effect+of+second%2Dlanguage+orthographic+input+on+the+phonological+encoding+of+Mandarin+words%2E+Applied+Psycholinguistics%2E)
8. Jonathan Clark, Dan Garrette, Iulia Turc, and John Wieting. 2021. [CANINE: Pre-training an efficient tokenization-free encoder for language representation](https://md.research.elamur.ai/free/en/papers/clark-2022-canine-pre-training-efficient-tokenization/). Transactions of the Association for Computational Linguistics, 10:73–91. https://doi.org/10.1162/tacl a 00448 [↗](https://doi.org/10.1162/tacl)
9. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: Pre-training text encoders as discriminators rather than generators. In Proceedings of ICLR. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Quoc+V%2E+Le%2C+and+Christopher+D%2E+Manning%2E+2020%2E+ELECTRA%3A+Pre%2Dtraining+text+encoders+as+discriminators+rather+than+generators%2E+In+Proceedings+of+ICLR%2E)
10. Florian Coulmas. 1991. The Writing Systems of the World. Blackwell Publishers. [find](https://scholar.google.com/scholar?q=Florian+Coulmas%2E+1991%2E+The+Writing+Systems+of+the+World%2E+Blackwell+Publishers%2E)
11. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. [DeBERTa: Decoding-enhanced BERT with disentangled attention](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/). In Proceedings of ICLR.
12. Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020. [OCNLI: Original Chinese natural language inference](https://md.research.elamur.ai/free/en/papers/hu-2020-ocnli-original-chinese-natural/). In Findings of EMNLP. https://doi.org/10.18653/v1/2020.findings -emnlp.314 [↗](https://doi.org/10.18653/v1/2020.findings)
13. Li Huang, Junjie Li, Weiwei Jiang, Zhiyu Zhang, Minchuan Chen, Shaojun Wang, and Jing Xiao. 2021. PHMOSpell: Phonological and morphological knowledge guided Chinese spelling check. In Proceedings of ACL, pages 5958–5967. https://doi.org/10 .18653/v1/2021.acl-long.464 [find](https://scholar.google.com/scholar?q=Li+Huang%2C+Junjie+Li%2C+Weiwei+Jiang%2C+Zhiyu+Zhang%2C+Minchuan+Chen%2C+Shaojun+Wang%2C+and+Jing+Xiao%2E+2021%2E+PHMOSpell%3A+Phonological+and+morphological+knowledge+guided+Chinese+spelling+check%2E+In+Proceedings+of+A)
14. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. [Revisiting pre-trained models for Chinese natural language processing](https://md.research.elamur.ai/free/en/papers/cui-2020-revisiting-pre-trained-models-chinese/). In Findings of EMNLP.
15. Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang, Shijin Wang, and Guoping Hu. 2019a. [Pre-training with whole word masking for Chinese BERT](https://md.research.elamur.ai/free/en/papers/cui-2019-pre-training-whole-word-masking/). IEEE/ACM TASLP, 29:3504–3514.
16. Mario Michael Krell, Matej Kosec, Sergio P. Perez, and Andrew Fitzgibbon. 2021. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint, abs/ 2107.02027. [find](https://scholar.google.com/scholar?q=Mario+Michael+Krell%2C+Matej+Kosec%2C+Sergio+P%2E+Perez%2C+and+Andrew+Fitzgibbon%2E+2021%2E+Efficient+sequence+packing+without+cross%2Dcontamination%3A+Accelerating+large+language+models+without+impacting+performance)
17. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019b. [A span-extraction dataset for Chinese machine reading comprehension](https://md.research.elamur.ai/free/en/papers/cui-2018-span-extraction-dataset-chinese/). In Proceedings of EMNLP-IJCNLP.
18. Taku Kudo. 2018. [Subword regularization: Improving neural network translation models with multiple subword candidates](https://md.research.elamur.ai/free/en/papers/kudo-2018-subword-regularization-improving-neural-network/). In Proceedings of ACL. https://doi.org/10.18653/v1/P18-1007 [↗](https://doi.org/10.18653/v1/P18-1007)
19. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of NAACL-HLT.
20. Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2020. ZEN: Pre-training Chinese text encoder enhanced by n-gram representations. In Findings of EMNLP. https://doi.org/10.18653/v1/2020 .findings-emnlp.425 [↗](https://doi.org/10.18653/v1/2020)
21. Chuanhai Dong, Jiajun Zhang, Chengqing Zong, Masanori Hattori, and Hui Di. 2016. [Character-based lstm-crf with radical-level features for chinese named entity recognition](https://md.research.elamur.ai/free/en/papers/kudo-2018-sentencepiece-simple-language-independent-subword/). In International Conference on Computer Pro-Taku Kudo and John Richardson. 2018. Sentence-Piece: A simple and language independe…
22. Yuxuan Lai, Yijia Liu, Yansong Feng, Songfang Huang, and Dongyan Zhao. 2021. Lattice-BERT: Leveraging multi-granularity representations in Chinese pre-trained language models. In Proceedings of NAACL-HLT. [find](https://scholar.google.com/scholar?q=Yuxuan+Lai%2C+Yijia+Liu%2C+Yansong+Feng%2C+Songfang+Huang%2C+and+Dongyan+Zhao%2E+2021%2E+Lattice%2DBERT%3A+Leveraging+multi%2Dgranularity+representations+in+Chinese+pre%2Dtrained+language+models%2E+In+Proceedings+of+NAACL%2D)
23. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. [ALBERT: A lite BERT for self-supervised learning of language representations](https://md.research.elamur.ai/free/en/papers/lan-2019-albert-lite-bert-self-supervised/). In Proceedings of ICLR.
24. Antoine Nzeyimana and Andre Niyongabo Rubungo. 2022. KinyaBERT: A morphology-aware kinyarwanda language model. In Proceedings of ACL. https://doi.org/10 .18653/v1/2022.acl-long.367 [find](https://scholar.google.com/scholar?q=Antoine+Nzeyimana+and+Andre+Niyongabo+Rubungo%2E+2022%2E+KinyaBERT%3A+A+morphology%2Daware+kinyarwanda+language+model%2E+In+Proceedings+of+ACL%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E18653%2Fv1%2F2022%2Eacl%2Dlong%2E367)
25. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. [find](https://scholar.google.com/scholar?q=Hector+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2012%2E+The+Winograd+schema+challenge%2E+In+Thirteenth+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2E)
26. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of NAACL-HLT. https://doi.org/10 .18653/v1/N18-1202
27. Jingyang Li and Maosong Sun. 2007. Scalable term selection for text categorization. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Jingyang+Li+and+Maosong+Sun%2E+2007%2E+Scalable+term+selection+for+text+categorization%2E+In+Proceedings+of+EMNLP%2E)
28. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and Korean voice search. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. https://doi.org/10.1109/ICASSP.2012 .6289079 — [doi:10.1109/icassp.2012](https://doi.org/10.1109/icassp.2012) [↗](https://doi.org/10.1109/ICASSP.2012)
29. Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li. 2019. Is word segmentation necessary for deep learning of Chinese representations? In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Xiaoya+Li%2C+Yuxian+Meng%2C+Xiaofei+Sun%2C+Qinghong+Han%2C+Arianna+Yuan%2C+and+Jiwei+Li%2E+2019%2E+Is+word+segmentation+necessary+for+deep+learning+of+Chinese+representations%3F+In+Proceedings+of+ACL%2E)
30. Zhongguo Li and Maosong Sun. 2009. Punctuation as implicit annotations for Chinese word segmentation. Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zhongguo+Li+and+Maosong+Sun%2E+2009%2E+Punctuation+as+implicit+annotations+for+Chinese+word+segmentation%2E+Computational+Linguistics%2E)
31. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of ACL. https://doi.org/10.18653/v1/P16-1162 [↗](https://doi.org/10.18653/v1/P16-1162)
32. Frederick Liu, Han Lu, Chieh Lo, and Graham Neubig. 2017. Learning character-level compositionality with visual features. In Proceedings of ACL, pages 2059–2068. [find](https://scholar.google.com/scholar?q=Frederick+Liu%2C+Han+Lu%2C+Chieh+Lo%2C+and+Graham+Neubig%2E+2017%2E+Learning+character%2Dlevel+compositionality+with+visual+features%2E+In+Proceedings+of+ACL%2C+pages+2059%E2%80%932068%2E)
33. Chenglei Si, Zhengyan Zhang, Yingfa Chen, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2023. READIN: A Chinese multi-task benchmark with realistic and diverse input noises. arXiv, abs/2302.07324. [find](https://scholar.google.com/scholar?q=Chenglei+Si%2C+Zhengyan+Zhang%2C+Yingfa+Chen%2C+Xiaozhi+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2023%2E+READIN%3A+A+Chinese+multi%2Dtask+benchmark+with+realistic+and+diverse+input+noises%2E+arXiv%2C+abs%2F2302%2E07324%2E)
34. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [RoBERTa: A robustly optimized BERT pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint, abs/1907.11692.
35. Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020. [Investigating prior knowledge for challenging Chinese machine reading comprehension](https://md.research.elamur.ai/free/en/papers/sun-2019-investigating-prior-knowledge-challenging/). Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl a 00305 [↗](https://doi.org/10.1162/tacl)
36. Wentao Ma, Yiming Cui, Chenglei Si, Ting Liu, Shijin Wang, and Guoping Hu. 2020. Char-BERT: Character-aware pre-trained language model. In Proceedings of COLING. [find](https://scholar.google.com/scholar?q=Wentao+Ma%2C+Yiming+Cui%2C+Chenglei+Si%2C+Ting+Liu%2C+Shijin+Wang%2C+and+Guoping+Hu%2E+2020%2E+Char%2DBERT%3A+Character%2Daware+pre%2Dtrained+language+model%2E+In+Proceedings+of+COLING%2E)
37. Maosong Sun, Xinxiong Chen, Kaixu Zhang, Zhipeng Guo, and Zhiyuan Liu. 2016. THU-LAC: An efficient lexical analyzer for Chinese. GitHub. [find](https://scholar.google.com/scholar?q=Maosong+Sun%2C+Xinxiong+Chen%2C+Kaixu+Zhang%2C+Zhipeng+Guo%2C+and+Zhiyuan+Liu%2E+2016%2E+THU%2DLAC%3A+An+efficient+lexical+analyzer+for+Chinese%2E+GitHub%2E)
38. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. [Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/). In Proceedings of ACL. https://doi.org/10.18653/v1/P19-1334 [↗](https://doi.org/10.18653/v1/P19-1334)
39. Yaming Sun, Lei Lin, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2014. Radical-enhanced chinese character embedding. In Proceedings of COLING, pages 279–286. [find](https://scholar.google.com/scholar?q=Yaming+Sun%2C+Lei+Lin%2C+Nan+Yang%2C+Zhenzhou+Ji%2C+and+Xiaolong+Wang%2E+2014%2E+Radical%2Denhanced+chinese+character+embedding%2E+In+Proceedings+of+COLING%2C+pages+279%E2%80%93286%2E)
40. Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for Chinese character representations. In Proceedings of NeurIPS. [find](https://scholar.google.com/scholar?q=Yuxian+Meng%2C+Wei+Wu%2C+Fei+Wang%2C+Xiaoya+Li%2C+Ping+Nie%2C+Fan+Yin%2C+Muyu+Li%2C+Qinghong+Han%2C+Xiaofei+Sun%2C+and+Jiwei+Li%2E+2019%2E+Glyce%3A+Glyph%2Dvectors+for+Chinese+character+representations%2E+In+Proceedings+of+NeurI)
41. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019. [ERNIE: Enhanced representation through knowledge integration](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-enhanced-representation-through-knowledge/). arXiv preprint, abs/ 1904.09223.
42. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Proceedings of NeurIPS, volume 26.
43. Zijun Sun, Xiaoya Li, Xiaofei Sun, Yuxian Meng, Xiang Ao, Qing He, Fei Wu, and Jiwei Li. 2021. ChineseBERT: Chinese pretraining enhanced by glyph and Pinyin information. In Proceedings of ACL, pages 2065–2075. https://doi.org/10.18653/v1/2021 .acl-long.161 [↗](https://doi.org/10.18653/v1/2021)
44. Yi Tay, Vinh Tran, Sebastian Ruder, Jai Gupta, Hyung Won Chung, Dara Bahri, Zhen Qin, Simon Baumgartner, Cong Yu, and Donald Metzler. 2022. [Charformer: Fast character transformers via gradient-based subword tokenization](https://md.research.elamur.ai/free/en/papers/tay-2021-charformer-fast-character-transformers-gradient/). In Proceedings of ICLR. standing evaluation benchmark. In Proceedings of COLING…
45. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2022. ByT5: Towards a token-free future with pre-trained byte-to-byte models. Transactions of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linting+Xue%2C+Aditya+Barua%2C+Noah+Constant%2C+Rami+Al%2DRfou%2C+Sharan+Narang%2C+Mihir+Kale%2C+Adam+Roberts%2C+and+Colin+Raffel%2E+2022%2E+ByT5%3A+Towards+a+token%2Dfree+future+with+pre%2Dtrained+byte%2Dto%2Dbyte+models%2E+Transac)
46. Jinxing Yu, Xun Jian, Hao Xin, and Yangqiu Song. 2017. Joint embeddings of Chinese words, characters, and fine-grained subchar-acter components. In Proceedings of EMNLP, pages 286–291. [find](https://scholar.google.com/scholar?q=Jinxing+Yu%2C+Xun+Jian%2C+Hao+Xin%2C+and+Yangqiu+Song%2E+2017%2E+Joint+embeddings+of+Chinese+words%2C+characters%2C+and+fine%2Dgrained+subchar%2Dacter+components%2E+In+Proceedings+of+EMNLP%2C+pages+286%E2%80%93291%2E)
47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of NeurIPS.
48. Xinsong Zhang, Pengshuai Li, and Hang Li. 2021a. AMBERT: A pre-trained language model with multi-grained tokenization. In Findings of ACL. https://doi.org/10 .18653/v1/2021.findings-acl.37 [find](https://scholar.google.com/scholar?q=Xinsong+Zhang%2C+Pengshuai+Li%2C+and+Hang+Li%2E+2021a%2E+AMBERT%3A+A+pre%2Dtrained+language+model+with+multi%2Dgrained+tokenization%2E+In+Findings+of+ACL%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E18653%2Fv1%2F2021%2Efindings%2Dacl%2E37)
49. Junqiu Wei, Qun Liu, Yinpeng Guo, and Xin Jiang. 2021. Training multilingual pre-trained language model with byte-level \| subwords. arXiv preprint, abs/2101.09469. — [arXiv:2101.09469](https://arxiv.org/abs/2101.09469) [find](https://scholar.google.com/scholar?q=Junqiu+Wei%2C+Qun+Liu%2C+Yinpeng+Guo%2C+and+Xin+Jiang%2E+2021%2E+Training+multilingual+pre%2Dtrained+language+model+with+byte%2Dlevel+%5C%7C+subwords%2E+arXiv+preprint%2C+abs%2F2101%2E09469%2E+%E2%80%94+%5BarXiv%3A2101%2E09469%5D%28https%3A%2F%2Farxiv%2E)
50. Yuan Zhang, Jason Baldridge, and Luheng He. 2019a. [PAWS: Paraphrase adversaries from word scrambling](https://md.research.elamur.ai/free/en/papers/zhang-2019-paws-paraphrase-adversaries-word-scrambling/). In Proceedings of NAACL-HLT.
51. Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. 2019. NEZHA: Neural contextualized representation for Chinese language understanding. arXiv, abs/1904.00204. — [arXiv:1904.00204](https://arxiv.org/abs/1904.00204) [find](https://scholar.google.com/scholar?q=Junqiu+Wei%2C+Xiaozhe+Ren%2C+Xiaoguang+Li%2C+Wenyong+Huang%2C+Yi+Liao%2C+Yasheng+Wang%2C+Jiashu+Lin%2C+Xin+Jiang%2C+Xiao+Chen%2C+and+Qun+Liu%2E+2019%2E+NEZHA%3A+Neural+contextualized+representation+for+Chinese+language+under)
52. Yun Zhang, Yongguo Liu, Jiajing Zhu, Ziqiang Zheng, Xiaofeng Liu, Weiguang Wang, Zijie Chen, and Shuangqing Zhai. 2019b. Learning chinese word embeddings from stroke, structure and pinyin of characters. In Proceedings of CIKM, pages 1011–1020. https://doi.org/10.1145/3357384.3358005 — [doi:10.1145/3357384.3358005](https://doi.org/10.1145/3357384.3358005) [↗](https://doi.org/10.1145/3357384.3358005)
53. Dongling Xiao, Yu-Kun Li, Han Zhang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. [ERNIE-Gram: Pre-training with explicitly n-gram masked language modeling for natural language understanding](https://md.research.elamur.ai/free/en/papers/xiao-2020-ernie-gram-pre-training/). In Proceedings of NAACL-HLT. https://doi.org/10.18653/v1/2021.naacl-main.136 [↗](https://doi.org/10.18653/v1/2021.naacl-main.136)
54. Zhengyan Zhang, Yuxian Gu, Xu Han, Shengqi Chen, Chaojun Xiao, Zhenbo Sun, Yuan Yao, Fanchao Qi, Jian Guan, Pei Ke, Yanzheng Cai, Guoyang Zeng, Zhixing Tan, Zhiyuan Liu, Minlie Huang, Wentao Han, Yang Liu, Xiaoyan Zhu, and Maosong Sun. 2021b. [CPM-2: Large-scale cost-effective pre-trained language mo…](https://md.research.elamur.ai/free/en/papers/zhang-2021-cpm-2-large-scale/)
55. Liang Xu, Qianqian Dong, Cong Yu, Yin Tian, Weitang Liu, Lu Li, and Xuanwei Zhang. 2020a. [CLUENER2020: Fine-grained name entity recognition for Chinese](https://md.research.elamur.ai/free/en/papers/xu-2020-cluener2020-fine-grained-named/). arXiv preprint, abs/2001.04351.
56. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipe… — [CPM: A Large-scale Generative Chinese Pre-trained Language Model](https://md.research.elamur.ai/free/en/papers/zhang-2020-cpm-large-scale-generative/)
57. AI Open. https://doi.org/10.1016/j .aiopen.2021.07.001 https://doi.org/10.18653/v1/P19 -1075 — [doi:10.1016/j](https://doi.org/10.1016/j) [find](https://scholar.google.com/scholar?q=AI+Open%2E+https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2Fj+%2Eaiopen%2E2021%2E07%2E001+https%3A%2F%2Fdoi%2Eorg%2F10%2E18653%2Fv1%2FP19+%2D1075+%E2%80%94+%5Bdoi%3A10%2E1016%2Fj%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E1016%2Fj%29)
58. Chujie Zheng, Minlie Huang, and Aixin Sun. 2019. [ChID: A large-scale Chinese IDiom dataset for cloze test](https://md.research.elamur.ai/free/en/papers/zheng-2019-chid-large-scale-chinese/). In Proceedings of ACL.
59. Wei Zhu. 2020. MVP-BERT: Redesigning vocabularies for Chinese BERT and multi-vocab pretraining. arXiv preprint, abs/2011.08539. — [arXiv:2011.08539](https://arxiv.org/abs/2011.08539) [find](https://scholar.google.com/scholar?q=Wei+Zhu%2E+2020%2E+MVP%2DBERT%3A+Redesigning+vocabularies+for+Chinese+BERT+and+multi%2Dvocab+pretraining%2E+arXiv+preprint%2C+abs%2F2011%2E08539%2E+%E2%80%94+%5BarXiv%3A2011%2E08539%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2011%2E08539%29)

*Updated 31 October 2026*