---
title: "Applying Conditional Random Fields to Japanese — Kudo 2004"
description: "Japanese is written without spaces, so a morphological analyzer has to decide word boundaries and part-of-speech tags together."
authors: ["Taku Kudo", "Kaoru Yamamoto", "Yuji Matsumoto"]
year: 2004
venue: "Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/W04-3230/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kudo-2004-applying-conditional-random-fields-japanese/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kudo-2004-applying-conditional-random-fields-japanese/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2004](https://md.research.elamur.ai/free/en/papers/2004/) / Applying Conditional Random Fields to Japanese Morphological Analysis

# Applying Conditional Random Fields to Japanese Morphological Analysis

**Authors:** [Taku Kudo](https://md.research.elamur.ai/free/en/authors/taku-kudo/), Kaoru Yamamoto, [Yuji Matsumoto](https://md.research.elamur.ai/free/en/authors/yuji-matsumoto/) · **Year:** 2004 · **Venue:** Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/W04-3230/) · [PDF](https://aclanthology.org/W04-3230.pdf) · [OpenAlex](https://openalex.org/W165283731)

**Citations:** 710 citations (openalex-snapshot · 2026-10-14) · [199](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Japanese is written without spaces, so a morphological analyzer has to decide word boundaries and part-of-speech tags together. Earlier CRF work assumed the token sequence was already fixed, so CRFs could not be applied directly. Kudo, Yamamoto and Matsumoto reformulate CRFs over a lexicon-derived word lattice rather than character-level B/I tagging. In their version, candidate paths may contain different numbers of tokens, and one globally normalized model scores each complete path. This lets them combine many overlapping features: levels of the hierarchical POS tagset, conjugation form and type, affixes and character types. Generative HMMs struggle to use such features. The model also avoids the label and length biases that push MEMMs toward low-entropy or short paths. On two standard annotated corpora, KC and RWCP, using the same data as earlier HMM and MEMM studies, CRFs achieve the best segmentation and tagging scores. The approach shows that discriminative sequence models can exploit a lexicon, and it carries over to other unsegmented languages such as Chinese or Thai.

## Key points
- On KC, L2-CRFs score F=98.96 for segmentation, 98.31 for top-level POS and 96.75 for full tags. MEMMs (Uchimoto 01) reach 96.44/95.81/94.27, rule-based JUMAN 98.70/98.09/93.73, and a bigram HMM 96.22/94.96/91.85.
- On RWCP, L2-CRFs score 99.11/98.73/97.66, ahead of the extended HMMs of Asahara 00 (98.87/98.33/96.95).
- Length bias in MEMMs shows up in the error counts. MEMMs output too-long tokens 416 times, which is 70% of their errors. CRFs do so 79 times and HMMs 306 times.
- L1-regularized CRFs are slightly less accurate but use about 1/8–1/6 as many active features: 90,163 vs 791,798 on KC and 101,757 vs 580,032 on RWCP.
- The IPA tagset used by ChaSen has roughly 500 distinct labels when hierarchy branches are counted. CRF feature templates cover these levels without the hand-tuned smoothing that HMMs require.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Computing & GPUs](https://md.research.elamur.ai/free/en/shelves/computing/)

Topics: [Part-of-speech tagging](https://md.research.elamur.ai/free/en/topics/part-of-speech-tagging/)

## Cited by

Cited by

199 papers of our library cite it (of the 140884 whose bibliographies we read); 7 of them have a page here, 192 not yet.

- [A Probabilistic Morphological Analyzer for Syriac](https://md.research.elamur.ai/free/en/papers/aclbulk-2010-mcclanahan-probabilistic-morphological-analyzer-syriac/)
- [Extracting Semantic Transfer Rules from Parallel Corpora with SMT Phrase Aligners](https://md.research.elamur.ai/free/en/papers/haugereid-2012-extracting-semantic-transfer-rules-parallel/)
- [Intra-Sentential Subject Zero Anaphora Resolution using Multi-Column Convolutional Neural Network](https://md.research.elamur.ai/free/en/papers/iida-2016-intra-sentential-subject-zero-anaphora/)
- [JGLUE: Japanese General Language Understanding Evaluation](https://md.research.elamur.ai/free/en/papers/kurihara-2022-jglue-japanese-general-language-understanding/)
- [Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis](https://md.research.elamur.ai/free/en/papers/neubig-2011-pointwise-prediction-robust-adaptable-japanese/)
- [Joint English Spelling Error Correction and POS Tagging for Language Learners Writing](https://md.research.elamur.ai/free/en/papers/sakaguchi-2012-joint-english-spelling-error-correction/)
- [Construction of English MWE Dictionary and its Application to POS Tagging](https://md.research.elamur.ai/free/en/papers/shigeto-2013-construction-english-mwe-dictionary-its/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kudo2004applying,
  title = {{Applying Conditional Random Fields to Japanese Morphological Analysis}},
  author = {Kudo, Taku and Yamamoto, Kaoru and Matsumoto, Yuji},
  booktitle = {Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing},
  year = {2004},
  url = {https://aclanthology.org/W04-3230/}
}
```

- [← previous Annotating Discourse Connectives and Their Arguments](https://md.research.elamur.ai/free/en/papers/miltsakaki-2004-annotating-discourse-connectives-their-arguments/)
- [next → Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics](https://md.research.elamur.ai/free/en/papers/lin-2004-automatic-evaluation-machine-translation-quality/)
- [↑ back to papers of 2004](https://md.research.elamur.ai/free/en/papers/2004/)

## References

18 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Masayuki Asahara and Yuji Matsumoto. 2000. Extended models and tools for high-performance part-of-speech tagger. In Proc of COLING, pages 21–27. [find](https://scholar.google.com/scholar?q=Masayuki+Asahara+and+Yuji+Matsumoto%2E+2000%2E+Extended+models+and+tools+for+high%2Dperformance+part%2Dof%2Dspeech+tagger%2E+In+Proc+of+COLING%2C+pages+21%E2%80%9327%2E)
2. Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ci You Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(6):1190–1208. [find](https://scholar.google.com/scholar?q=Richard+H%2E+Byrd%2C+Peihuang+Lu%2C+Jorge+Nocedal%2C+and+Ci+You+Zhu%2E+1995%2E+A+limited+memory+algorithm+for+bound+constrained+optimization%2E+SIAM+Journal+on+Scientific+Computing%2C+16%286%29%3A1190%E2%80%931208%2E)
3. Stanley F. Chen and Ronald. Rosenfeld. 1999. A gaussian prior for smoothing maximum entropy models. Technical report, Carnegie Mellon University. [find](https://scholar.google.com/scholar?q=Stanley+F%2E+Chen+and+Ronald%2E+Rosenfeld%2E+1999%2E+A+gaussian+prior+for+smoothing+maximum+entropy+models%2E+Technical+report%2C+Carnegie+Mellon+University%2E)
4. Joshua Goodman. 2004. Exponential priors for maximum entropy models. In Proc. of HLT/NAACL. [find](https://scholar.google.com/scholar?q=Joshua+Goodman%2E+2004%2E+Exponential+priors+for+maximum+entropy+models%2E+In+Proc%2E+of+HLT%2FNAACL%2E)
5. John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. of ICML, pages 282–289. [find](https://scholar.google.com/scholar?q=John+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+Proc%2E+of+ICML%2C+pages+282%E2%80%93289%2E)
6. Dong C. Liu and Jorge Nocedal. 1989. On the limited memory BFGS method for large scale optimization. Math. Programming, 45(3, (Ser. B)):503–528. [find](https://scholar.google.com/scholar?q=Dong+C%2E+Liu+and+Jorge+Nocedal%2E+1989%2E+On+the+limited+memory+BFGS+method+for+large+scale+optimization%2E+Math%2E+Programming%2C+45%283%2C+%28Ser%2E+B%29%29%3A503%E2%80%93528%2E)
7. Andrew McCallum and Wei Li. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In In Proc. of CoNLL. [find](https://scholar.google.com/scholar?q=Andrew+McCallum+and+Wei+Li%2E+2003%2E+Early+results+for+named+entity+recognition+with+conditional+random+fields%2C+feature+induction+and+web%2Denhanced+lexicons%2E+In+In+Proc%2E+of+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccallum-2003-early-results-named-entity-recognition/)
8. Andrew McCallum, Dayne Freitag, and Fernando Pereira. 2000. Maximum entropy markov models for information and segmentation. In Proc. of ICML, pages 591–598. [find](https://scholar.google.com/scholar?q=Andrew+McCallum%2C+Dayne+Freitag%2C+and+Fernando+Pereira%2E+2000%2E+Maximum+entropy+markov+models+for+information+and+segmentation%2E+In+Proc%2E+of+ICML%2C+pages+591%E2%80%93598%2E)
9. Andrew McCallum. 2003. Efficiently inducing features of conditional random fields. In Nineteenth Conference on Uncertainty in Artificial Intelligence (UAI03). [find](https://scholar.google.com/scholar?q=Andrew+McCallum%2E+2003%2E+Efficiently+inducing+features+of+conditional+random+fields%2E+In+Nineteenth+Conference+on+Uncertainty+in+Artificial+Intelligence+%28UAI03%29%2E)
10. Fuchun Peng and Andrew McCallum. 2004. Accurate information extraction from research papers. In Proc. of HLT/NAACL. [find](https://scholar.google.com/scholar?q=Fuchun+Peng+and+Andrew+McCallum%2E+2004%2E+Accurate+information+extraction+from+research+papers%2E+In+Proc%2E+of+HLT%2FNAACL%2E)
11. Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. Chinese segmentation and new word detection using conditional random fields (to appear). In Proc. of COLING. [find](https://scholar.google.com/scholar?q=Fuchun+Peng%2C+Fangfang+Feng%2C+and+Andrew+McCallum%2E+2004%2E+Chinese+segmentation+and+new+word+detection+using+conditional+random+fields+%28to+appear%29%2E+In+Proc%2E+of+COLING%2E) [our card](https://md.research.elamur.ai/free/en/papers/peng-2004-chinese-segmentation-new-word-detection/)
12. Simon Perkins, Kevin Lacker, and James Thiler. 2003. Grafting: Fast, incremental feature selection by gradient descent in function space. JMLR, 3:1333–1356. [find](https://scholar.google.com/scholar?q=Simon+Perkins%2C+Kevin+Lacker%2C+and+James+Thiler%2E+2003%2E+Grafting%3A+Fast%2C+incremental+feature+selection+by+gradient+descent+in+function+space%2E+JMLR%2C+3%3A1333%E2%80%931356%2E)
13. Della Pietra, Stephen, Vincent J. Della Pietra, and John D. Lafferty. 1997. Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(4):380–393. [find](https://scholar.google.com/scholar?q=Della+Pietra%2C+Stephen%2C+Vincent+J%2E+Della+Pietra%2C+and+John+D%2E+Lafferty%2E+1997%2E+Inducing+features+of+random+fields%2E+IEEE+Transactions+on+Pattern+Analysis+and+Machine+Intelligence%2C+19%284%29%3A380%E2%80%93393%2E)
14. David Pinto, Andrew McCallum, Xing Wei, and W. Bruce Croft. 2003. Table extraction using conditional random fields. In In Proc. of SIGIR, pages 235–242. [find](https://scholar.google.com/scholar?q=David+Pinto%2C+Andrew+McCallum%2C+Xing+Wei%2C+and+W%2E+Bruce+Croft%2E+2003%2E+Table+extraction+using+conditional+random+fields%2E+In+In+Proc%2E+of+SIGIR%2C+pages+235%E2%80%93242%2E)
15. Fei Sha and Fernando Pereira. 2003. Shallow parsing with conditional random fields. In Proc. of HLT-NAACL, pages 213–220. [find](https://scholar.google.com/scholar?q=Fei+Sha+and+Fernando+Pereira%2E+2003%2E+Shallow+parsing+with+conditional+random+fields%2E+In+Proc%2E+of+HLT%2DNAACL%2C+pages+213%E2%80%93220%2E)
16. Kiyotaka Uchimoto, Satoshi Sekine, and Hitoshi Isahara. 2001. The unknown word problem: a morphological analysis of Japanese using maximum entropy aided by a dictionary. In Proc. of EMNLP, pages 91–99. [find](https://scholar.google.com/scholar?q=Kiyotaka+Uchimoto%2C+Satoshi+Sekine%2C+and+Hitoshi+Isahara%2E+2001%2E+The+unknown+word+problem%3A+a+morphological+analysis+of+Japanese+using+maximum+entropy+aided+by+a+dictionary%2E+In+Proc%2E+of+EMNLP%2C+pages+91%E2%80%9399)
17. Kiyotaka Uchimoto, Chikashi Nobata, Atsushi Yamada, Satoshi Sekine, and Hitoshi Isahara. 2002. Morphological analysis of the spontaneous speech corpus. In Proc of COLING, pages 1298– 1302. [find](https://scholar.google.com/scholar?q=Kiyotaka+Uchimoto%2C+Chikashi+Nobata%2C+Atsushi+Yamada%2C+Satoshi+Sekine%2C+and+Hitoshi+Isahara%2E+2002%2E+Morphological+analysis+of+the+spontaneous+speech+corpus%2E+In+Proc+of+COLING%2C+pages+1298%E2%80%93+1302%2E)
18. Kiyotaka Uchimoto, Chikashi Nobata, Atsushi Yamada, and Hitoshi Isahara Satoshi Sekine. 2003. Morphological analysis of a large spontaneous speech corpus in Japanese. In Proc. of ACL, pages 479–488. [find](https://scholar.google.com/scholar?q=Kiyotaka+Uchimoto%2C+Chikashi+Nobata%2C+Atsushi+Yamada%2C+and+Hitoshi+Isahara+Satoshi+Sekine%2E+2003%2E+Morphological+analysis+of+a+large+spontaneous+speech+corpus+in+Japanese%2E+In+Proc%2E+of+ACL%2C+pages+479%E2%80%93488%2E)