---
title: "探索中文预训练模型的混合粒度编码和IDF遮蔽 — 邵云帆 2024"
description: "Most Chinese pre-trained language models encode text character by character."
authors: ["邵云帆", "孙天祥", "邱锡鹏"]
year: 2024
venue: "中文信息学报 (Journal of Chinese Information Processing)"
source: "http://jcip.cipsc.org.cn/article/id/zwxxxb_3663"
licence: "Free access (jcip.cipsc.org.cn), no open licence stated — readonly; © 中文信息学报"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/shao-2024-exploring-chinese-pre/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/shao-2024-exploring-chinese-pre/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2024](https://md.research.elamur.ai/free/papers/2024/page/4/) / 探索中文预训练模型的混合粒度编码和IDF遮蔽

# 探索中文预训练模型的混合粒度编码和IDF遮蔽

**Authors:** 邵云帆, 孙天祥, 邱锡鹏 · **Year:** 2024 · **Venue:** 中文信息学报 (Journal of Chinese Information Processing) · **Licence:** Free access (jcip.cipsc.org.cn), no open licence stated — readonly; © 中文信息学报

**Links:** [Publisher's page](http://jcip.cipsc.org.cn/article/id/zwxxxb_3663) · [PDF](http://jcip.cipsc.org.cn/cn/article/pdf/preview/zwxxxb_3663.pdf)

## Summary
Most Chinese pre-trained language models encode text character by character. This produces long input sequences and high Transformer cost. Word-level encoding shortens the input, but it depends on an external segmenter and suffers from out-of-vocabulary words and sparse data. Shao, Sun and Qiu (Fudan University) propose a BERT-base-style model with a mixed-grained vocabulary, in which characters, words and phrases coexist. The vocabulary is learned directly from the pre-training corpus with a SentencePiece unigram language model, and sentences are tokenized by Viterbi decoding. Any unseen word falls back to single characters, so OOV is avoided.

The larger vocabulary has a long-tailed token distribution, which could slow training. To compensate, the authors introduce IDF-masking: the model preferentially masks rare, informative words chosen by corpus-level inverse document frequency rather than masking at random.

The model was trained from scratch on 13GB of open data. It matches or beats BERT, RoBERTa and multi-granularity models such as AMBERT, LICHEE-BERT and Lattice-BERT on CLUE and other Chinese benchmarks. It also shortens inputs and speeds up inference, especially for long texts.

## Key points
- **Vocabulary comparison:**
  - The mixed-grained vocabulary has 60k entries, about 58k of them Chinese tokens. Chinese BERT has 20k entries, about 7k Chinese.
  - Encoded-to-raw length ratio falls from 0.969 with BERT to 0.565, measured on 5,000 sampled sentences.
- **Pre-training data and setup:**
  - Corpus: 13GB, about 9M documents. SentencePiece was trained on 20M sampled sentences.
  - IDF lexicon: about 1M words, with words of frequency below 10 removed.
  - Model and schedule: BERT-base architecture, 1M steps, batch size 384, maximum length 512. Training took roughly 7 days on 8 RTX3090 GPUs.
- **How IDF-masking works:**
  - A conventional segmenter splits each sentence into words.
  - The top 30% of words by IDF form a candidate pool.
  - 15% of the sentence's words are sampled from this pool for masking.
  - Masking is applied to the contiguous mixed-grained tokens that cover each chosen word.
- **Results:**
  - Mixed-grained encoding with IDF-masking reaches 76.13% average accuracy on the CLUE dev set.
  - In the ablation, IDF-masking beats whole-word masking for both character-level and mixed-grained models, with the larger gain for the mixed-grained one.
  - Best (mean) test accuracies: ChnSentiCorp 96.2 (96.0), THUCNews 98.0 (97.9), LCQMC 87.7 (87.5).
- **Limitation:** The authors leave tasks that need fine-grained, segmentation-aligned outputs to future work. These include POS tagging, NER and span-based reading comprehension.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/shelves/nlp/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{2024,
  title = {{探索中文预训练模型的混合粒度编码和IDF遮蔽}},
  author = {邵云帆 and 孙天祥 and 邱锡鹏},
  journal = {中文信息学报 (Journal of Chinese Information Processing)},
  year = {2024},
  url = {http://jcip.cipsc.org.cn/article/id/zwxxxb_3663}
}
```

- [← previous 大语言模型评测综述](https://md.research.elamur.ai/free/papers/luo-2024-evaluating-large-language/)
- [next → 故事启发大语言模型的时序知识图谱预测](https://md.research.elamur.ai/free/papers/chen-2024-temporal-knowledge-graph/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/papers/2024/page/4/)