---
title: "Learning Question Classifiers — paper by Li and Roth (2002)"
description: "Li and Roth instead propose a two-level taxonomy with 6 coarse and 50 fine answer classes. 19 references, cited by 419 papers here."
authors: ["Xin Li", "Dan Roth"]
year: 2002
venue: "COLING 2002: The 19th International Conference on Computational Linguistics"
source: "https://aclanthology.org/C02-1150/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2002](https://md.research.elamur.ai/free/en/papers/2002/) / Learning Question Classifiers

# Learning Question Classifiers

**Authors:** [Xin Li](https://md.research.elamur.ai/free/en/authors/xin-li/) and [Dan Roth](https://md.research.elamur.ai/free/en/authors/dan-roth/) · **Year:** 2002 · **Venue:** [COLING 2002: The 19th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2002](https://md.research.elamur.ai/free/en/venues/coling/2002/)) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/C02-1150/) · [PDF](https://aclanthology.org/C02-1150.pdf) · [OpenAlex](https://openalex.org/W2070246124)

**Citations:** 1,234 citations (OpenAlex, 2026-10-14) · [419](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

The paper treats question classification as a learning problem in its own right. The goal is to predict the semantic type of the answer a factual question expects, so that a downstream question answering system can discard unsuitable candidates and choose type-specific search strategies. Earlier TREC systems mostly used hand-written rules with a coarse inventory of roughly 20 types or fewer. Li and Roth instead propose a two-level taxonomy with 6 coarse and 50 fine answer classes. They train a pair of chained classifiers built on the SNoW architecture with Winnow. The coarse prediction narrows the candidate label set for the fine stage. Because many questions are genuinely ambiguous, the system may return several labels per question. With 5,500 training questions and the 500 TREC 10 questions as a test set, the full feature set (including semantically related word lists) scores 98.8% on coarse and 95% on fine classes when up to five labels are allowed. The results show that fine-grained answer typing can be learned accurately, and that semantic features matter much more than surface ones.

## Key points

- **Taxonomy:** six coarse classes (ABBREVIATION, ENTITY, DESCRIPTION, HUMAN, LOCATION, NUMERIC VALUE) split into 50 non-overlapping fine classes.
- **Data:** training on 5,500 labelled questions; evaluation on the 500 TREC 10 questions.
- **Features:** added cumulatively in the order Word, Pos, Chunk, NE, Head, RelWord.
  - Fine-class accuracy with up to five labels rises from 86.0% (words only) to 95.0% (with RelWord).
  - The jump from RelWord is much larger for fine classes than for coarse ones.
- **Single-label setting:** accuracy with one label is 91.0% for coarse and 84.2% for fine classes.
  - On average the coarse classifier outputs 1.54 labels and the fine classifier 2.05.
- **Flat vs. hierarchical:** a flat classifier over all 50 fine classes performs about as well as the hierarchical one. The authors therefore find no real benefit from the coarse layer.

## Interesting ideas
- **Multiple labels for ambiguous inputs:** the classifier returns a small set of plausible labels, obtained by thresholding probabilities derived from its activations. Downstream components keep every reasonable reading at little cost, which suits any pipeline where an early hard decision would spread errors.
- **Label filtering between stages:** the coarse stage's surviving classes are expanded into the confusion set for the fine stage. This is a cheap way to impose a label hierarchy on any multi-class problem.
- **Feature types instead of hand-listed features:** a handful of feature types are defined and then expanded from the data into a very large feature space. This avoids writing rules by hand and makes retraining on a new taxonomy fast, which is helpful when adapting to a new language or domain.
- **Semantic word-class features:** lists of semantically related words give the biggest gain, especially for fine classes. Lexical-semantic resources can therefore matter more than extra syntactic layers.
- **Checking whether a hierarchy helps:** the authors measure normalised pairwise confusion between fine classes and see that errors do not cluster inside coarse classes. This diagnostic is worth running before assuming a hierarchy will help a classifier.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 108 works · 67,432 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

## Appears in

Agglomerations: [🏙️ Champaign–Urbana](https://md.research.elamur.ai/free/en/agglomerations/champaign/)

## Cite

Cite as: Xin Li and Dan Roth. 2002. Learning Question Classifiers. COLING 2002: The 19th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2002learning,
  title = {{Learning Question Classifiers}},
  author = {Xin Li and Dan Roth},
  booktitle = {COLING 2002: The 19th International Conference on Computational Linguistics},
  year = {2002},
  doi = {10.3115/1072228.1072378},
  url = {https://doi.org/10.3115/1072228.1072378}
}
```

- [← previous Automatic Labeling of Semantic Roles](https://md.research.elamur.ai/free/en/papers/gildea-2002-automatic-labeling-semantic-roles/)
- [next → Multiword Expressions: A Pain in the Neck for NLP](https://md.research.elamur.ai/free/en/papers/sag-2002-multiword-expressions-pain-neck-nlp/)
- [↑ back to papers of 2002](https://md.research.elamur.ai/free/en/papers/2002/)

## References

19 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. P. Abney. 1991. Parsing by chunks. In S. P. Abney R. C. Berwick and C. Tenny, editors, Principle-based parsing: Computation and Psycholinguistics, pages 257–278. Kluwer, Dordrecht. [find](https://scholar.google.com/scholar?q=S%2E+P%2E+Abney%2E+1991%2E+Parsing+by+chunks%2E+In+S%2E+P%2E+Abney+R%2E+C%2E+Berwick+and+C%2E+Tenny%2C+editors%2C+Principle%2Dbased+parsing%3A+Computation+and+Psycholinguistics%2C+pages+257%E2%80%93278%2E+Kluwer%2C+Dordrecht%2E)
2. A. Carlson, C. Cumby, J. Rosen, and D. Roth. 1999. The SNoW learning architecture. Technical Report UIUCDCS-R-99-2101, UIUC Computer Science Department, May. [find](https://scholar.google.com/scholar?q=A%2E+Carlson%2C+C%2E+Cumby%2C+J%2E+Rosen%2C+and+D%2E+Roth%2E+1999%2E+The+SNoW+learning+architecture%2E+Technical+Report+UIUCDCS%2DR%2D99%2D2101%2C+UIUC+Computer+Science+Department%2C+May%2E)
3. C. Cumby and D. Roth. 2000. Relational representations that facilitate learning. In Proc. of the International Conference on the Principles of Knowledge Representation and Reasoning, pages 425–434. [find](https://scholar.google.com/scholar?q=C%2E+Cumby+and+D%2E+Roth%2E+2000%2E+Relational+representations+that+facilitate+learning%2E+In+Proc%2E+of+the+International+Conference+on+the+Principles+of+Knowledge+Representation+and+Reasoning%2C+pages+425%E2%80%93434%2E)
4. Y. Even-Zohar and D. Roth. 2001. A sequential model for multi class classification. In EMNLP-2001, the SIGDAT Conference on Empirical Methods in Natural Language Processing, pages 10–19. [find](https://scholar.google.com/scholar?q=Y%2E+Even%2DZohar+and+D%2E+Roth%2E+2001%2E+A+sequential+model+for+multi+class+classification%2E+In+EMNLP%2D2001%2C+the+SIGDAT+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+10%E2%80%9319%2E)
5. A. R. Golding and D. Roth. 1999. A Winnow based approach to context-sensitive spelling correction. Machine Learning, 34(1-3):107–130. [find](https://scholar.google.com/scholar?q=A%2E+R%2E+Golding+and+D%2E+Roth%2E+1999%2E+A+Winnow+based+approach+to+context%2Dsensitive+spelling+correction%2E+Machine+Learning%2C+34%281%2D3%29%3A107%E2%80%93130%2E)
6. S. Harabagiu, D. Moldovan, M. Pasca, R. Mihalcea, M. Surdeanu, R. Bunescu, R. Girju, V. Rus, and P. Morarescu. 2001. Falcon: Boosting knowledge for answer engines. In Proceedings of the 9th Text Retrieval Conference, NIST. [find](https://scholar.google.com/scholar?q=S%2E+Harabagiu%2C+D%2E+Moldovan%2C+M%2E+Pasca%2C+R%2E+Mihalcea%2C+M%2E+Surdeanu%2C+R%2E+Bunescu%2C+R%2E+Girju%2C+V%2E+Rus%2C+and+P%2E+Morarescu%2E+2001%2E+Falcon%3A+Boosting+knowledge+for+answer+engines%2E+In+Proceedings+of+the+9th+Text+Retri)
7. U. Hermjakob. 2001. Parsing and question classification for question answering. In ACL-2001 Workshop on Open-Domain Question Answering. [find](https://scholar.google.com/scholar?q=U%2E+Hermjakob%2E+2001%2E+Parsing+and+question+classification+for+question+answering%2E+In+ACL%2D2001+Workshop+on+Open%2DDomain+Question+Answering%2E)
8. L. Hirschman, M. Light, E. Breck, and J. Burger. 1999. Deep read: A reading comprehension system. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=L%2E+Hirschman%2C+M%2E+Light%2C+E%2E+Breck%2C+and+J%2E+Burger%2E+1999%2E+Deep+read%3A+A+reading+comprehension+system%2E+In+Proceedings+of+the+37th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2E)
9. E. Hovy, L. Gerber, U. Hermjakob, C. Lin, and D. Ravichandran. 2001. Toward semantics-based answer pinpointing. In Proceedings of the DARPA Human Language Technology conference (HLT). San Diego, CA. [our card](https://md.research.elamur.ai/free/en/papers/hovy-2001-semantics-based-answer-pinpointing/)
10. A. Ittycheriah, M. Franz, W-J Zhu, A. Ratnaparkhi, and R.J. Mammone. 2001. IBM’s statistical question answering system. In Proceedings of the 9th Text Retrieval Conference, NIST. [find](https://scholar.google.com/scholar?q=A%2E+Ittycheriah%2C+M%2E+Franz%2C+W%2DJ+Zhu%2C+A%2E+Ratnaparkhi%2C+and+R%2EJ%2E+Mammone%2E+2001%2E+IBM%E2%80%99s+statistical+question+answering+system%2E+In+Proceedings+of+the+9th+Text+Retrieval+Conference%2C+NIST%2E)
11. W. G. Lehnert. 1986. A conceptual theory of question answering. In B. J. Grosz, K. Sparck Jones, and B. L. Webber, editors, Natural Language Processing, pages 651–657. Kaufmann, Los Altos, CA. [find](https://scholar.google.com/scholar?q=W%2E+G%2E+Lehnert%2E+1986%2E+A+conceptual+theory+of+question+answering%2E+In+B%2E+J%2E+Grosz%2C+K%2E+Sparck+Jones%2C+and+B%2E+L%2E+Webber%2C+editors%2C+Natural+Language+Processing%2C+pages+651%E2%80%93657%2E+Kaufmann%2C+Los+Altos%2C+CA%2E)
12. X. Li and D. Roth. 2001. Exploring evidence for shallow parsing. In Proc. of the Annual Conference on Computational Natural Language Learning. [find](https://scholar.google.com/scholar?q=X%2E+Li+and+D%2E+Roth%2E+2001%2E+Exploring+evidence+for+shallow+parsing%2E+In+Proc%2E+of+the+Annual+Conference+on+Computational+Natural+Language+Learning%2E)
13. M. Light, G. Mann, E. Riloff, and E. Breck. 2001. Analyses for Elucidating Current Question Answering Technology. Journal for Natural Language Engineering. forthcoming. [find](https://scholar.google.com/scholar?q=M%2E+Light%2C+G%2E+Mann%2C+E%2E+Riloff%2C+and+E%2E+Breck%2E+2001%2E+Analyses+for+Elucidating+Current+Question+Answering+Technology%2E+Journal+for+Natural+Language+Engineering%2E+forthcoming%2E)
14. V. Punyakanok and D. Roth. 2001. The use of classifiers in sequential inference. In NIPS-13; The 2000 Conference on Advances in Neural Information Processing Systems, pages 995–1001. MIT Press. [find](https://scholar.google.com/scholar?q=V%2E+Punyakanok+and+D%2E+Roth%2E+2001%2E+The+use+of+classifiers+in+sequential+inference%2E+In+NIPS%2D13%3B+The+2000+Conference+on+Advances+in+Neural+Information+Processing+Systems%2C+pages+995%E2%80%931001%2E+MIT+Press%2E)
15. D. Roth and W. Yih. 2001. Relational learning via propositional algorithms: An information extraction case study. In Proc. of the International Joint Conference on Artificial Intelligence, pages 1257–1263. [find](https://scholar.google.com/scholar?q=D%2E+Roth+and+W%2E+Yih%2E+2001%2E+Relational+learning+via+propositional+algorithms%3A+An+information+extraction+case+study%2E+In+Proc%2E+of+the+International+Joint+Conference+on+Artificial+Intelligence%2C+pages+1257%E2%80%93)
16. D. Roth, G. Kao, X. Li, R. Nagarajan, V. Punyakanok, N. Rizzolo, W. Yih, C. O. Alm, and L. G. Moran. 2002. Learning components for a question answering system. In TREC-2001. [find](https://scholar.google.com/scholar?q=D%2E+Roth%2C+G%2E+Kao%2C+X%2E+Li%2C+R%2E+Nagarajan%2C+V%2E+Punyakanok%2C+N%2E+Rizzolo%2C+W%2E+Yih%2C+C%2E+O%2E+Alm%2C+and+L%2E+G%2E+Moran%2E+2002%2E+Learning+components+for+a+question+answering+system%2E+In+TREC%2D2001%2E)
17. D. Roth. 1998. Learning to resolve natural language ambiguities: A unified approach. In Proc. of the American Association of Artificial Intelligence, pages 806– 813. [find](https://scholar.google.com/scholar?q=D%2E+Roth%2E+1998%2E+Learning+to+resolve+natural+language+ambiguities%3A+A+unified+approach%2E+In+Proc%2E+of+the+American+Association+of+Artificial+Intelligence%2C+pages+806%E2%80%93+813%2E)
18. A. Singhal, S. Abney, M. Bacchiani, M. Collins, D. Hindle, and F. Pereira. 2000. AT&T at TREC-8. In Proceedings of the 8th Text Retrieval Conference, NIST. [find](https://scholar.google.com/scholar?q=A%2E+Singhal%2C+S%2E+Abney%2C+M%2E+Bacchiani%2C+M%2E+Collins%2C+D%2E+Hindle%2C+and+F%2E+Pereira%2E+2000%2E+AT%26T+at+TREC%2D8%2E+In+Proceedings+of+the+8th+Text+Retrieval+Conference%2C+NIST%2E)
19. E. Voorhees. 2000. Overview of the TREC-9 question answering track. In The Ninth Text Retrieval Conference (TREC-9), pages 71–80. NIST SP 500-249. [find](https://scholar.google.com/scholar?q=E%2E+Voorhees%2E+2000%2E+Overview+of+the+TREC%2D9+question+answering+track%2E+In+The+Ninth+Text+Retrieval+Conference+%28TREC%2D9%29%2C+pages+71%E2%80%9380%2E+NIST+SP+500%2D249%2E)

*Updated 24 October 2026*