---
title: "The First International Chinese Word Segmentation Bakeoff (2003)"
description: "Chinese word segmentation research has been hard to compare because each group tested on its… 12 references, cited by 75 papers here."
authors: ["Richard Sproat", "Thomas Emerson"]
year: 2003
venue: "Proceedings of the Second SIGHAN Workshop on Chinese Language Processing"
source: "https://aclanthology.org/W03-1719/"
doi: "10.3115/1119250.1119269"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sproat-2003-first-international-chinese-word-segmentation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sproat-2003-first-international-chinese-word-segmentation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2003](https://md.research.elamur.ai/free/en/papers/2003/) / The First International Chinese Word Segmentation Bakeoff

# The First International Chinese Word Segmentation Bakeoff

**Authors:** [Richard Sproat](https://md.research.elamur.ai/free/en/authors/richard-sproat/) and [Thomas Emerson](https://md.research.elamur.ai/free/en/authors/thomas-emerson/) · **Year:** 2003 · **Venue:** [Proceedings of the Second SIGHAN Workshop on Chinese Language Processing](https://md.research.elamur.ai/free/en/venues/sighan-workshop-on-chinese-language-processing/) ([SIGHAN Workshop on Chinese Language Processing 2003](https://md.research.elamur.ai/free/en/venues/sighan-workshop-on-chinese-language-processing/2003/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W03-1719/) · [PDF](https://aclanthology.org/W03-1719.pdf) · [DOI 10.3115/1119250.1119269](https://doi.org/10.3115/1119250.1119269) · [OpenAlex](https://openalex.org/W2033295622)

**Citations:** 211 citations (OpenAlex, 2026-10-14) · [75](https://md.research.elamur.ai/free/en/papers/sproat-2003-first-international-chinese-word-segmentation/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Chinese word segmentation research has been hard to compare because each group tested on its own data and several incompatible segmentation standards exist. The authors ran the first international, SIGHAN-sponsored bakeoff in 2003 to fix this. They gave participants four corpora, each following a different standard: Academia Sinica (AS), the Penn Chinese Treebank (CTB), Hong Kong CityU (HK) and Beijing University (PK). Each corpus had a closed track (only its own training data allowed) and an open track (any extra resources, with their contribution disclosed). Twelve sites from six regions took part. Scoring was automatic and strict: only the exact segmentation of the given standard counted, with no credit for plausible alternatives. Systems were compared against a maximum-matching baseline built from the training lexicon and a "cheating" topline built from the test lexicon. Results came as recall, precision, F, and recall on OOV and in-vocabulary words, with confidence values. No single system won across all tracks. The bakeoff matters because it created a shared, reusable benchmark and evaluation protocol for the field.

## Key points
- Corpus sizes in training/test words were AS 5.8M/12K, CTB 250K/40K, HK 240K/35K and PK 1.1M/17K. The corpora used different encodings: Big Five, Big Five with HKSCS characters, EUC-CN (GB 2312-80) and GBK (CP936).
- The 12 participating sites came from Mainland China (2), Hong Kong (3), Japan, Singapore, Taiwan and the US (4). Each site chose its own corpora and tracks and could not be tested on data from its own institution.
- The timetable was tight. Training data was released March 15, test data April 22, and segmented output was due April 25. Scoring used a simple Perl script built on `diff -y`.
- Example results:
  - PK closed was led by S01 with F = 0.951 (8 systems).
  - CTB open was led by S02 with F = 0.912.
  - OOV recall varied far more across systems than in-vocabulary recall did.
- Participants reported annotation inconsistencies during the contest. These were partly corrected in CTB, but the AS report came too late to fix.

## Interesting ideas
- **Paired baseline and "oracle" topline:** The same simple algorithm is run twice, once with a lexicon from the training data and once with a lexicon taken from the test data. This brackets realistic performance and can be reused for any lexicon-dependent task.
- **Closed vs. open tracks with disclosure:** Separating "own training data only" from "anything goes" makes clear how much of a score comes from the method and how much from extra resources. Asking open-track teams to attribute their gains, for example to a dictionary, makes the comparison more honest.
- **Splitting recall into OOV and in-vocabulary parts:** A single F score hides how a system handles unseen words. Reporting the two recalls separately shows which systems generalize, which helps any tokenization or tagging benchmark.
- **Strict single-standard scoring:** The organizers refused to judge which alternative segmentations were acceptable. This keeps the evaluation objective and simple, and lets multiple standards be compared side by side instead of being merged.
- **Planned benchmark extensions:** The authors recommend short query-like test strings and recording whether each system is commercial. Both would show gaps between lab benchmarks and real deployment, such as search queries or the commercial-versus-research quality lag.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [AT&T](https://md.research.elamur.ai/free/en/companies/at-t/) — Company, Bedminster, United States · 29 works · 15,393 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [✂️ Tokenization & segmentation](https://md.research.elamur.ai/free/en/topics/tokenization/) · 77 works

## Cite

Cite as: Richard Sproat and Thomas Emerson. 2003. The First International Chinese Word Segmentation Bakeoff. Proceedings of the Second SIGHAN Workshop on Chinese Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sproat2003first,
  title = {{The First International Chinese Word Segmentation Bakeoff}},
  author = {Richard Sproat and Thomas Emerson},
  booktitle = {Proceedings of the Second SIGHAN Workshop on Chinese Language Processing},
  year = {2003},
  doi = {10.3115/1119250.1119269},
  url = {https://doi.org/10.3115/1119250.1119269}
}
```

- [← previous Syntax-based Alignment of Multiple Translations: Extracting Paraphrases and Generating New Sentences](https://md.research.elamur.ai/free/en/papers/pang-2003-syntax-based-alignment-multiple-translations/)
- [next → Sentence Alignment for Monolingual Comparable Corpora](https://md.research.elamur.ai/free/en/papers/barzilay-2003-sentence-alignment-monolingual-comparable-corpora/)
- [↑ back to papers of 2003](https://md.research.elamur.ai/free/en/papers/2003/)

## References

12 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. GB/T 13715–92. 1993. Contemporary Chinese language word-segmentation specification for information processing. Technical report, , Beijing. c  0.0021 0.0031 0.0032 [find](https://scholar.google.com/scholar?q=GB%2FT+13715%E2%80%9392%2E+1993%2E+Contemporary+Chinese+language+word%2Dsegmentation+specification+for+information+processing%2E+Technical+report%2C+%2C+Beijing%2E+c+%18+0%2E0021+0%2E0031+0%2E0032)
2. Site S08 S03 S11 word count 34,955 34,955 34,955 R 0.958 0.909 0.898 Table 11: Scores for HK open, sorted by F. c  0.0029 0.0029 0.0037 0.0038 0.0036 0.0045 Site S10 S01 S08 S04 S03 S11 word count 17,194 17,194 17,194 17,194 17,194 17,194 R 0.963 0.963 0.939 0.933 0.940 0.905 Table 12: Scores for P… [find](https://scholar.google.com/scholar?q=Site+S08+S03+S11+word+count+34%2C955+34%2C955+34%2C955+R+0%2E958+0%2E909+0%2E898+Table+11%3A+Scores+for+HK+open%2C+sorted+by+F%2E+c+%18+0%2E0029+0%2E0029+0%2E0037+0%2E0038+0%2E0036+0%2E0045+Site+S10+S01+S08+S04+S03+S11+word+count+17)
3. Charles Grinstead and J. Laurie Snell. 1997. Introduction to Probability. American Mathematical Society, Providence, RI, 2nd edition. [find](https://scholar.google.com/scholar?q=Charles+Grinstead+and+J%2E+Laurie+Snell%2E+1997%2E+Introduction+to+Probability%2E+American+Mathematical+Society%2C+Providence%2C+RI%2C+2nd+edition%2E)
4. Chu-Ren Huang, Keh-Jiann Chen, Chang Lili, and Fengyi Chen. 1997. Segmentation standard for Chinese natural language processing. International Journal of Computational Linguistics and Chinese Language Processing, 2(2):47–62. [find](https://scholar.google.com/scholar?q=Chu%2DRen+Huang%2C+Keh%2DJiann+Chen%2C+Chang+Lili%2C+and+Fengyi+Chen%2E+1997%2E+Segmentation+standard+for+Chinese+natural+language+processing%2E+International+Journal+of+Computational+Linguistics+and+Chinese+Language)
5. Alan Smeaton and Ross Wilkinson. 1997. Spanish and chinese document retrieval at TREC-5. In E.M. Vorhees and D.K. Harman, editors, Proceedings of the Fifth Text REtrieval Conference. [find](https://scholar.google.com/scholar?q=Alan+Smeaton+and+Ross+Wilkinson%2E+1997%2E+Spanish+and+chinese+document+retrieval+at+TREC%2D5%2E+In+E%2EM%2E+Vorhees+and+D%2EK%2E+Harman%2C+editors%2C+Proceedings+of+the+Fifth+Text+REtrieval+Conference%2E)
6. Richard Sproat and Chilin Shih. 2001. Corpus-based methods in Chinese morphology and phonology. Technical report, Linguistic Society of America Summer Institute, Santa Barbara, CA. [http://www.research.att.com/˜rws/newindex/notes.pdf](http://www.research.att.com/˜rws/newindex/notes.pdf). [↗](http://www.research.att.com/˜rws/newindex/notes.pdf)
7. Yongheng Wang, Haiju Su, and Yan Mo. 1990. Automatic processing of Chinese words. Journal of Chinese Information Processing, 4(4):1–11. [find](https://scholar.google.com/scholar?q=Yongheng+Wang%2C+Haiju+Su%2C+and+Yan+Mo%2E+1990%2E+Automatic+processing+of+Chinese+words%2E+Journal+of+Chinese+Information+Processing%2C+4%284%29%3A1%E2%80%9311%2E)
8. Ross Wilkinson. 1998. Chinese document retrieval at TREC-6. In E.M. Vorhees and D.K. Harman, editors, Proceedings of the Sixth Text REtrieval Conference. [find](https://scholar.google.com/scholar?q=Ross+Wilkinson%2E+1998%2E+Chinese+document+retrieval+at+TREC%2D6%2E+In+E%2EM%2E+Vorhees+and+D%2EK%2E+Harman%2C+editors%2C+Proceedings+of+the+Sixth+Text+REtrieval+Conference%2E)
9. Zimin Wu and Gwyneth Tseng. 1993. Chinese text segmentation for text retrieval: Achievements and problems. Journal of the American Society for Information Science, 44(9):532–542. [find](https://scholar.google.com/scholar?q=Zimin+Wu+and+Gwyneth+Tseng%2E+1993%2E+Chinese+text+segmentation+for+text+retrieval%3A+Achievements+and+problems%2E+Journal+of+the+American+Society+for+Information+Science%2C+44%289%29%3A532%E2%80%93542%2E)
10. Andi Wu. 2003. Customizable segmentation of morphologically derived words in Chinese. International Journal of Computational Linguistics and Chinese Language Processing, 8(1): forthcoming. c  0.0022 0.0037 0.0037 [find](https://scholar.google.com/scholar?q=Andi+Wu%2E+2003%2E+Customizable+segmentation+of+morphologically+derived+words+in+Chinese%2E+International+Journal+of+Computational+Linguistics+and+Chinese+Language+Processing%2C+8%281%29%3A+forthcoming%2E+c+%18+0%2E0022+)
11. R 
 0.788 0.579 0.616 R   0.971 0.935 0.920 P 0.954 0.863 0.860 F 0.956 0.886 0.879 OOV 0.071 0.071 0.071 R 
 0.799 0.743 0.675 0.712 0.647 0.503 R   0.975 0.980 0.959 0.949 0.962 0.934 c  0.0031 0.0035 0.0037 0.0036 0.0043 0.0051 OOV 0.069 0.069 0.069 0.069 0.069 0.069 P 0.956 0.943 0.938… [find](https://scholar.google.com/scholar?q=R+%01%03%02+0%2E788+0%2E579+0%2E616+R+%04+%02+0%2E971+0%2E935+0%2E920+P+0%2E954+0%2E863+0%2E860+F+0%2E956+0%2E886+0%2E879+OOV+0%2E071+0%2E071+0%2E071+R+%01%03%02+0%2E799+0%2E743+0%2E675+0%2E712+0%2E647+0%2E503+R+%04+%02+0%2E975+0%2E980+0%2E959+0%2E949+0%2E962+0%2E934+c+%18+0%2E)
12. Fei Xia. 1999. Segmentation guideline, Chinese Treebank Project. Technical report, University of Pennsylvania. [http://morph.ldc.upenn.edu/ctb/](http://morph.ldc.upenn.edu/ctb/). [↗](http://morph.ldc.upenn.edu/ctb/)

*Updated 24 October 2026*