--- title: "The Third International Chinese Language — Levow 2006" description: "This paper reports on the third SIGHAN bakeoff (Spring 2006)." authors: ["Gina-Anne Levow"] year: 2006 venue: "Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing" source: "https://aclanthology.org/W06-0115/" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/levow-2006-third-international-chinese-language-processing/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2006](https://md.research.elamur.ai/free/en/papers/2006/) / The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition # The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition **Authors:** [Gina-Anne Levow](https://md.research.elamur.ai/free/en/authors/gina-anne-levow/) · **Year:** 2006 · **Venue:** Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/W06-0115/) · [PDF](https://aclanthology.org/W06-0115.pdf) · [OpenAlex](https://openalex.org/W2252066972) **Citations:** 281 citations (openalex-snapshot · 2026-10-14) · [108](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary This paper reports on the third SIGHAN bakeoff (Spring 2006). It is a shared evaluation of Chinese word segmentation (WS) and, for the first time, Chinese named entity recognition (NER). NER was added because names are a major source of out-of-vocabulary (OOV) words and are poorly covered by lexicons. The organizers collected five corpora from MSRA, UPUC, LDC, CITYU and CKIP, in both Simplified and Traditional characters. They converted all data to common formats and ran open and closed tracks under a tight test schedule. Scoring was automatic, with confidence intervals for comparing systems. Twenty-nine groups submitted 144 runs. WS systems reached up to 0.979 F (MSRA open), and NER systems reached about 0.91 F (MSRA open). Results varied strongly across corpora, mainly because of OOV rates and register shifts. The bakeoff shows that OOV handling, annotation consistency, encodings and mixed-script text remain open problems. It also provides benchmarks that later Chinese segmentation and NER work was measured against. ## Key points - **Participation:** 36 sites registered and 29 submitted results, giving 144 official runs (101 for WS, 43 for NER). The largest groups of teams came from the PRC (11) and Taiwan (7). - **Corpora:** - WS used CITYU, CKIP, MSRA and UPUC. - NER used CITYU, LDC and MSRA. - Training sizes ranged from 509K words (UPUC) to 5.5M words (CKIP). - NER covered PER, LOC and ORG entities; GPE appeared only in the LDC data. - **Baseline and topline:** The WS baseline was left-to-right maximal match using the training vocabulary. The topline ran the same algorithm with the test vocabulary. - Baseline F was 0.906 for CITYU, 0.892 for CKIP, 0.924 for MSRA and 0.828 for UPUC. - The best CITYU runs reached 99% of the topline F-score. - **UPUC difficulty:** UPUC had an OOV rate of 0.088, more than twice that of any other corpus. Its training data came only from the Chinese Treebank, while its test data also drew on other newswire and broadcast sources. - **Best NER F-scores:** - 0.9118 on MSRA open - 0.8651 on MSRA closed - 0.8903 on CITYU closed - 0.7627 on LDC closed The NER baseline (maximal match with tags that are unique in training) scored only about 0.43–0.53 F. ## Interesting ideas - **Baseline and topline from one simple algorithm.** Running dictionary maximal match once with the training vocabulary and once with the test vocabulary gives cheap reference points for each corpus. These points show how hard a dataset is, independent of any system, so results can be compared fairly across corpora of different difficulty. - **Open vs. closed tracks.** Separating systems that use only the provided training data from those that may add external lexicons, gazetteers or taggers shows how much outside resources actually help. This is especially relevant for OOV recall. - **Significance from binomial confidence intervals.** Treating recall or precision as a Bernoulli probability gives a 95% interval of roughly ±2·sqrt(p(1−p)/n). Two systems count as different only if their intervals do not overlap. This is an easy sanity check for any leaderboard. - **Normalizing everything into one format.** All data was converted to UTF-8 plus a CoNLL-style character/tag two-column format, using shared scripts. The scoring scripts were then released to participants. This lowers the barrier to entry and makes follow-up experiments reproducible. - **Reporting OOV and in-vocabulary recall separately.** Splitting recall into R_oov and R_iv shows where systems actually fail, for example under a register shift between training and test data. A single F-score hides this. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/) ## Cited by Cited by 108 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 100 not yet. - [Optimizing Chinese Word Segmentation for Machine Translation Performance](https://md.research.elamur.ai/free/en/papers/chang-2008-optimizing-chinese-word-segmentation-machine/) - [Emerging Cross-lingual Structure in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/conneau-2020-emerging-cross-lingual-structure-pretrained/) - [Dice Loss for Data-imbalanced NLP Tasks](https://md.research.elamur.ai/free/en/papers/li-2020-dice-loss-data-imbalanced-nlp/) - [A Unified MRC Framework for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/li-2020-unified-mrc-framework-named-entity/) - [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://md.research.elamur.ai/free/en/papers/liu-2023-summary-chatgpt-related-research-perspective/) - [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://md.research.elamur.ai/free/en/papers/sun-2019-ernie-2-0-continual/) - [ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT](https://md.research.elamur.ai/free/en/papers/wei-2023-chatie-zero-shot-information-extraction/) - [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{levow2006third, title = {{The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition}}, author = {Levow, Gina-Anne}, booktitle = {Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing}, year = {2006}, url = {https://aclanthology.org/W06-0115/} } ``` - [← previous The stages of event extraction](https://md.research.elamur.ai/free/en/papers/ahn-2006-stages-event-extraction/) - [next → Tips and Tricks of the Prague Arabic Dependency Treebank](https://md.research.elamur.ai/free/en/papers/aclbulk-2006-smrz-tips-tricks-prague-arabic-dependency/) - [↑ back to papers of 2006](https://md.research.elamur.ai/free/en/papers/2006/) ## References 4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Thomas Emerson. 2005. The Second International Chinese Word Segmentation Bakeoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, Jeju Island, Republic of Korea. [find](https://scholar.google.com/scholar?q=Thomas+Emerson%2E+2005%2E+The+Second+International+Chinese+Word+Segmentation+Bakeoff%2E+In+Proceedings+of+the+Fourth+SIGHAN+Workshop+on+Chinese+Language+Processing%2C+Jeju+Island%2C+Republic+of+Korea%2E) 2. Charles M. Grinstead and J. Laurie Snell. 1997. Introduction to Probability. American Mathematical Society, Providence, RI. [find](https://scholar.google.com/scholar?q=Charles+M%2E+Grinstead+and+J%2E+Laurie+Snell%2E+1997%2E+Introduction+to+Probability%2E+American+Mathematical+Society%2C+Providence%2C+RI%2E) 3. Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition. In Proceedings of the 6th Conference on Natural Language Learning 2002 (CoNLL-2002). [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang%2E+2002%2E+Introduction+to+the+CoNLL%2D2002+Shared+Task%3A+Language%2DIndependent+Named+Entity+Recognition%2E+In+Proceedings+of+the+6th+Conference+on+Natural+Language+Learning+2002+%28CoNLL%2D2) [our card](https://md.research.elamur.ai/free/en/papers/sang-2002-introduction-conll-2002-shared-task/) 4. Richard Sproat and Thomas Emerson. 2003. The First International Chinese Word Segmentation Bakeoff. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, Sapporo, Japan. [find](https://scholar.google.com/scholar?q=Richard+Sproat+and+Thomas+Emerson%2E+2003%2E+The+First+International+Chinese+Word+Segmentation+Bakeoff%2E+In+Proceedings+of+the+Second+SIGHAN+Workshop+on+Chinese+Language+Processing%2C+Sapporo%2C+Japan%2E)