---
title: "Neural Network Approach to Word Category Prediction for"
description: "The authors build NETgram, a feed-forward net that reads one-hot Brown Corpus part-of-speech… 6 references, cited by 16 papers here."
authors: ["Masami Nakamura", "Katsuteru Maruyama", "Takeshi Kawabata", "Kiyohiro Shikano"]
year: 1990
venue: "COLING 1990 Volume 3: Papers presented to the 13th International Conference on Computational Linguistics"
source: "https://aclanthology.org/C90-3038/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nakamura-1990-neural-network-approach-word-category/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nakamura-1990-neural-network-approach-word-category/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [1990](https://md.research.elamur.ai/free/en/papers/1990/) / Neural Network Approach to Word Category Prediction for English Texts

# Neural Network Approach to Word Category Prediction for English Texts

**Authors:** Masami Nakamura, Katsuteru Maruyama, Takeshi Kawabata and Kiyohiro Shikano · **Year:** 1990 · **Venue:** [COLING 1990 Volume 3: Papers presented to the 13th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 1990](https://md.research.elamur.ai/free/en/venues/coling/1990/)) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/C90-3038/) · [PDF](https://aclanthology.org/C90-3038.pdf) · [OpenAlex](https://openalex.org/W2105402874)

**Citations:** 35 citations (OpenAlex, 2026-10-14) · [16](https://md.research.elamur.ai/free/en/papers/nakamura-1990-neural-network-approach-word-category/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Isolated-word speech recognition makes many errors on short function words. The paper asks whether a small neural network can supply local syntactic expectations as well as a count-based n-gram while needing fewer parameters and less data. The authors build NETgram, a feed-forward net that reads one-hot Brown Corpus part-of-speech categories and outputs scores for the category of the next word. It has two narrow hidden layers and is trained with back-propagation. They first train a Bigram version and then grow it into Trigram and 4-gram versions by adding input blocks, starting from the bigram weights. On category prediction, NETgram roughly matches statistical models. It also gives sensible estimates for sequences never seen in training, and its lower hidden layer groups categories into linguistically coherent clusters. When its scores are used to rescore HMM word recognition, accuracy rises from 81.0% to 86.9%. With small training sets it beats a statistical trigram. This matters because it shows learned dense codes can smooth sparse symbolic statistics.

## Key points
- The architecture uses 89 input units (one per Brown Corpus category), two hidden layers of 16 units each, and 89 output units. Higher-order models add an input block wired to an extra lower hidden block.
- To speed up Bigram training, targets were full next-category distributions computed from 1,024 sentences (about 24,000 words). Trigram and 4-gram training used the actual next category as the target.
- The authors clustered lower-hidden-layer vectors by cosine similarity with a threshold of 0.985. This produced HAVE-verb, BE-verb, subject-pronoun and pre-noun groups.
- Data needs differ sharply between the two approaches:
  - A NETgram 4-gram trained on 2,048 sentences barely beat its trigram trained on 1,024.
  - A statistical 4-gram needed more than 6,000 sentences to reach that same NETgram trigram level.
- HMM word recognition was tested on 187 sentences (1,347 words) of keyboard conversations, spoken word by word by one male speaker:
  - HMM alone reached 81.0%.
  - NETgram reached 86.3% when trained on 512 sentences and 86.9% on 1,024, ahead of the statistical trigram.
  - At 30,000 sentences the statistical model edged ahead, 87.7% versus 87.2%.

## Interesting ideas
- Grow a higher-order model from a trained lower-order one by adding input blocks and reusing the learned weights as initialization; this avoids training each context size from scratch.
- Train on the empirical next-label distribution instead of single one-hot targets, which gives faster and more stable convergence when the same input maps to many possible outputs.
- Inspect a bottleneck layer by clustering its vectors and checking that the groups line up with known linguistic classes; this is a cheap test of whether the model learned structure rather than noise.
- Factor a word-level probability into a class-level sequence probability times a per-word lexical ratio P(w)/P(C), so that sparse word contexts borrow strength from category contexts.
- Rescore an n-best candidate list by combining an acoustic or primary score with a linguistic score using a tunable weight, then pick the best sequence with dynamic programming.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (tree): gparse could be warm-started this way when widening context features, for example by first training on a narrow window and then attaching extra blocks for more distant tokens.
- **I2** (learn): when several witness parsers disagree on silver data, their vote distribution could serve as a soft target rather than a forced single label.
- **I3** (tokens): cluster tagger hidden states per UPOS/XPOS tag and flag tags that fall outside their expected group as an explainable sanity check.
- **I4** (tokens): the tagger or lemmatizer could score rare words through their category trigram plus a dictionary ratio, keeping each score decomposable and explainable.
- **I5** (answer): candidate readings or answers could be reranked by mixing parser confidence with a frame-plausibility score, using a weight tuned on school-test data.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 2,263 works

## Cite

Cite as: Masami Nakamura, Katsuteru Maruyama, Takeshi Kawabata and Kiyohiro Shikano. 1990. Neural Network Approach to Word Category Prediction for English Texts. COLING 1990 Volume 3: Papers presented to the 13th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nakamura1990neural,
  title = {{Neural Network Approach to Word Category Prediction for English Texts}},
  author = {Masami Nakamura, Katsuteru Maruyama, Takeshi Kawabata and Kiyohiro Shikano},
  booktitle = {COLING 1990 Volume 3: Papers presented to the 13th International Conference on Computational Linguistics},
  year = {1990},
  doi = {10.3115/991146.991184},
  url = {https://doi.org/10.3115/991146.991184}
}
```

- [← previous Solving Thematic Divergences in Machine Translation](https://md.research.elamur.ai/free/en/papers/dorr-1990-solving-thematic-divergences-machine-translation/)
- [next → Machine Translation without a source text](https://md.research.elamur.ai/free/en/papers/somers-1990-machine-translation-without-source-text/)
- [↑ back to papers of 1990](https://md.research.elamur.ai/free/en/papers/1990/)

## References

6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [11 I,'.Jetinek, "Continuous Speech Recognition by Statistical Methods", Proceedings of the iI'~EI'], Vol.64, No.4 (1976.4) 121 K.Shikano, "hnprovenmnt of Word Recognition Result by Trigram Model", [CASSP 87, 29.2(1987.4) [find](https://scholar.google.com/scholar?q=%5B11+I%2C%27%2EJetinek%2C+%22Continuous+Speech+Recognition+by+Statistical+Methods%22%2C+Proceedings+of+the+iI%27%7EEI%27%5D%2C+Vol%2E64%2C+No%2E4+%281976%2E4%29+121+K%2EShikano%2C+%22hnprovenmnt+of+Word+Recognition+Result+by+Trigram+Model%22%2C+%5BC)
2. [al T.J.Sejnowski, C.R.Rosenberg, "NETtalk, A Parallel Network that l,earns to Read Aloud", Teeh. Report, The Johns llopkins University EESS -86-01 (1986) [find](https://scholar.google.com/scholar?q=%5Bal+T%2EJ%2ESejnowski%2C+C%2ER%2ERosenberg%2C+%22NETtalk%2C+A+Parallel+Network+that+l%2Cearns+to+Read+Aloud%22%2C+Teeh%2E+Report%2C+The+Johns+llopkins+University+EESS+%2D86%2D01+%281986%29)
3. [41 M.Nakamura, K.Shikano, "A Study of English Word Category Prediction Based on Neural Networks", ICASSP 89, SI 3.10(1989.5) 151 13town University, "Brown Corpus", Tech: Report, Brown University (1967) 16l D.E.Rumelhart, G.E.tllnton, R.d.Williams, "Parallel Distributed Processing", M.I.T. [find](https://scholar.google.com/scholar?q=%5B41+M%2ENakamura%2C+K%2EShikano%2C+%22A+Study+of+English+Word+Category+Prediction+Based+on+Neural+Networks%22%2C+ICASSP+89%2C+SI+3%2E10%281989%2E5%29+151+13town+University%2C+%22Brown+Corpus%22%2C+Tech%3A+Report%2C+Brown+University+%28196)
4. [7] F.Jelinek, R.Mercer, "Interpolated Estimation of Marker Source Parameters from Sparse Data", Pattern Recognition in Practice, ed. E,S. Gelsema and L. N. Kanal, North ttolland (1980) [find](https://scholar.google.com/scholar?q=%5B7%5D+F%2EJelinek%2C+R%2EMercer%2C+%22Interpolated+Estimation+of+Marker+Source+Parameters+from+Sparse+Data%22%2C+Pattern+Recognition+in+Practice%2C+ed%2E+E%2CS%2E+Gelsema+and+L%2E+N%2E+Kanal%2C+North+ttolland+%281980%29)
5. [81 S.E.Levinson, L.R.Rabiner, M.M.Sondhi, "An I n t r o d u c t i o n to the A p p l i c a t i o n of the Theory of Probabilistie Functions of a Marker Process to Automatic Speech Recognition", Bell Syst. Teeh. J. 62(4), pp1035-1074 (1983) [find](https://scholar.google.com/scholar?q=%5B81+S%2EE%2ELevinson%2C+L%2ER%2ERabiner%2C+M%2EM%2ESondhi%2C+%22An+I+n+t+r+o+d+u+c+t+i+o+n+to+the+A+p+p+l+i+c+a+t+i+o+n+of+the+Theory+of+Probabilistie+Functions+of+a+Marker+Process+to+Automatic+Speech+Recognition%22%2C+Bell+)
6. [91 T.1Ianazawa, G.Kawabata, K.Shikano, "Study of Separate Vector Q u a n t i z a t i o n for HMM P h o n e m e Recognition", ASJ 2-P-8(1988.10) [find](https://scholar.google.com/scholar?q=%5B91+T%2E1Ianazawa%2C+G%2EKawabata%2C+K%2EShikano%2C+%22Study+of+Separate+Vector+Q+u+a+n+t+i+z+a+t+i+o+n+for+HMM+P+h+o+n+e+m+e+Recognition%22%2C+ASJ+2%2DP%2D8%281988%2E10%29)

*Updated 30 October 2026*