--- title: "Learning surface text patterns for a — Ravichandran 2002" description: "Open-domain QA systems usually depend on heavy external resources such as named entity taggers, WordNet, parsers and ontologies." authors: ["Deepak Ravichandran", "Eduard Hovy"] year: 2002 venue: "Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P02-1006/" doi: "10.3115/1073083.1073092" licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ravichandran-2002-learning-surface-text-patterns-question/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ravichandran-2002-learning-surface-text-patterns-question/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2002](https://md.research.elamur.ai/free/en/papers/2002/) / Learning surface text patterns for a Question Answering System # Learning surface text patterns for a Question Answering System **Authors:** [Deepak Ravichandran](https://md.research.elamur.ai/free/en/authors/deepak-ravichandran/), [Eduard Hovy](https://md.research.elamur.ai/free/en/authors/eduard-hovy/) · **Year:** 2002 · **Venue:** Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/) **Links:** [Publisher's page](https://aclanthology.org/P02-1006/) · [PDF](https://aclanthology.org/P02-1006.pdf) · [DOI 10.3115/1073083.1073092](https://doi.org/10.3115/1073083.1073092) · [OpenAlex](https://openalex.org/W2167435923) **Citations:** 809 citations (openalex-snapshot · 2026-10-14) · [161](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Open-domain QA systems usually depend on heavy external resources such as named entity taggers, WordNet, parsers and ontologies. Yet the best system at TREC-10 relied mainly on a large inventory of surface patterns. This paper tests how far such patterns can go when they are learned automatically rather than written by hand. For each question type, the authors start from about ten seed question–answer pairs (e.g. Mozart / 1756) and query AltaVista with both terms. They keep only sentences that mention both terms and run them through a suffix tree to enumerate every repeated substring. The surviving strings become regular expressions with \ and \ slots. A second retrieval round, using only the question term, estimates each pattern's precision by cross-checking it on the other seed examples. At answer time, matching patterns extract candidates, which are ranked by pattern precision. On six TREC-10 question types, scored with MRR, web-sourced answers clearly beat answers drawn from the TREC corpus. The method's near-absence of language-specific tooling makes it attractive for rapid multilingual QA. ## Key points - **Question types and data.** Six types were taken from the Webclopedia QA Typology: BIRTHDATE, LOCATION, INVENTOR, DISCOVERER, DEFINITION and WHY-FAMOUS. Each seed query to AltaVista retrieves the top 1000 documents. - **Precision estimate.** Precision is computed as P = Ca / Co. Co counts sentences where the pattern matches with any word in the answer slot. Ca counts sentences where the slot holds the correct answer. Only patterns that match more than 5 examples are retained. - **Example precisions.** For BIRTHDATE, the pattern "\ ( \ - )" scores 1.0, while the looser "\ was born in \" scores 0.6. The system returns the top 5 answers after removing duplicates. - **MRR on the TREC corpus:** - BIRTHYEAR: 0.48 (8 questions) - INVENTOR: 0.17 (6 questions) - DISCOVERER: 0.13 (4 questions) - DEFINITION: 0.34 (102 questions) - WHY-FAMOUS: 0.33 (3 questions) - LOCATION: 0.75 (16 questions) Web retrieval performed notably better. The authors explain this by the web's redundancy: a short pattern usually matches somewhere, without needing long-distance dependencies. - **Reported weaknesses:** - no semantic or POS typing of the answer slot; - a single anchor term per pattern; - a fixed 50-byte answer window; - no case sensitivity; - name and date variants must be listed manually. ## Interesting ideas - **Suffix trees for pattern mining.** Suffix trees find repeated substrings of any length in linear time. Because overlapping variants are stored separately, a specific pattern and its more general substring each get their own precision. This is useful for any template-mining task where the right pattern length is unknown. - **Cross-example precision checks.** Patterns learned from one seed pair are scored on documents retrieved for different seeds of the same type. This filters out artifacts caused by mirrored or duplicated web pages, a cheap safeguard for any noisy web-bootstrapped resource. - **Precision as a simple likelihood estimate.** Comparing "slot matches anything" against "slot matches the gold answer" gives a quick reliability score for extraction rules. Those scores can then rank candidate answers directly, with no extra model. - **Seed-based bootstrapping instead of annotation.** A handful of seed pairs plus a search engine replaces hand-tagged corpora. The authors stress that this makes porting to a new language fast, which suits lower-resource settings such as Ukrainian. - **Suggested extensions.** The authors propose bounding the answer slot by 1–2 syntactic chunks instead of raw bytes. They also suggest requiring several question terms as anchors, which would prune many false candidates. Both are simple constraints worth adding to pattern-based extractors. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 161 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 153 not yet. - [A Survey of Paraphrasing and Textual Entailment Methods](https://md.research.elamur.ai/free/en/papers/androutsopoulos-2009-survey-paraphrasing-textual-entailment-methods/) - [Open Information Extraction from the Web](https://md.research.elamur.ai/free/en/papers/banko-2007-open-information-extraction-web/) - [The Distributional Inclusion Hypotheses and Lexical Entailment](https://md.research.elamur.ai/free/en/papers/geffet-2005-distributional-inclusion-hypotheses-lexical-entailment/) - [Discovering Relations among Named Entities from Large Corpora](https://md.research.elamur.ai/free/en/papers/hasegawa-2004-discovering-relations-among-named-entities/) - [How Can We Know What Language Models Know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) - [Generating Phrasal and Sentential Paraphrases: A Survey of Data-Driven Methods](https://md.research.elamur.ai/free/en/papers/madnani-2010-generating-phrasal-sentential-paraphrases-survey/) - [Espresso: Leveraging Generic Patterns for Automatically Harvesting Semantic Relations](https://md.research.elamur.ai/free/en/papers/pantel-2006-espresso-leveraging-generic-patterns-automatically/) - [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ravichandran2002learning, title = {{Learning surface text patterns for a Question Answering System}}, author = {Ravichandran, Deepak and Hovy, Eduard}, booktitle = {Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics}, year = {2002}, doi = {10.3115/1073083.1073092}, url = {https://doi.org/10.3115/1073083.1073092} } ``` - [← previous Learning Question Classifiers](https://md.research.elamur.ai/free/en/papers/li-2002-learning-question-classifiers/) - [next → Machine-learned contexts for linguistic operations in German sentence realization](https://md.research.elamur.ai/free/en/papers/gamon-2002-machine-learned-contexts-linguistic-operations/) - [↑ back to papers of 2002](https://md.research.elamur.ai/free/en/papers/2002/) ## References 14 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Brill, E., J. Lin, M. Banko, S. Dumais, and A. Ng. 2001. Data-Intensive Question Answering. Proceedings of the TREC-10 Conference. NIST, Gaithersburg, MD, 183–189. [find](https://scholar.google.com/scholar?q=Brill%2C+E%2E%2C+J%2E+Lin%2C+M%2E+Banko%2C+S%2E+Dumais%2C+and+A%2E+Ng%2E+2001%2E+Data%2DIntensive+Question+Answering%2E+Proceedings+of+the+TREC%2D10+Conference%2E+NIST%2C+Gaithersburg%2C+MD%2C+183%E2%80%93189%2E) 2. Gusfield, D. 1997. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Chapter 6: Linear Time construction of Suffix trees, 94–121. [find](https://scholar.google.com/scholar?q=Gusfield%2C+D%2E+1997%2E+Algorithms+on+Strings%2C+Trees+and+Sequences%3A+Computer+Science+and+Computational+Biology%2E+Chapter+6%3A+Linear+Time+construction+of+Suffix+trees%2C+94%E2%80%93121%2E) 3. Harabagiu, S., D. Moldovan, M. Pasca, R. Mihalcea, M. Surdeanu, R. Buneascu, R. Gîrju, V. Rus and P. Morarescu. 2001. FALCON: Boosting Knowledge for Answer Engines. Proceedings of the 9 th Text Retrieval Conference (TREC-9), NIST, 479–488. [find](https://scholar.google.com/scholar?q=Harabagiu%2C+S%2E%2C+D%2E+Moldovan%2C+M%2E+Pasca%2C+R%2E+Mihalcea%2C+M%2E+Surdeanu%2C+R%2E+Buneascu%2C+R%2E+G%C3%AErju%2C+V%2E+Rus+and+P%2E+Morarescu%2E+2001%2E+FALCON%3A+Boosting+Knowledge+for+Answer+Engines%2E+Proceedings+of+the+9+th+Text+Retrie) 4. Hovy, E.H., U. Hermjakob, and C.-Y. Lin. 2001. The Use of External Knowledge in Factoid QA. Proceedings of the TREC-10 Conference. NIST, Gaithersburg, MD, 166– 174. [find](https://scholar.google.com/scholar?q=Hovy%2C+E%2EH%2E%2C+U%2E+Hermjakob%2C+and+C%2E%2DY%2E+Lin%2E+2001%2E+The+Use+of+External+Knowledge+in+Factoid+QA%2E+Proceedings+of+the+TREC%2D10+Conference%2E+NIST%2C+Gaithersburg%2C+MD%2C+166%E2%80%93+174%2E) 5. Hovy, E.H., U. Hermjakob, and D. Ravichandran. 2002a. A Question/Answer Typology with Surface Text Patterns. Proceedings of the Human Language Technology (HLT) conference. San Diego, CA. [find](https://scholar.google.com/scholar?q=Hovy%2C+E%2EH%2E%2C+U%2E+Hermjakob%2C+and+D%2E+Ravichandran%2E+2002a%2E+A+Question%2FAnswer+Typology+with+Surface+Text+Patterns%2E+Proceedings+of+the+Human+Language+Technology+%28HLT%29+conference%2E+San+Diego%2C+CA%2E) 6. Hovy, E.H., U. Hermjakob, C.-Y. Lin, and D. Ravichandran. 2002b. Using Knowledge to Facilitate Pinpointing of Factoid Answers. Proceedings of the COLING-2002 conference. Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Hovy%2C+E%2EH%2E%2C+U%2E+Hermjakob%2C+C%2E%2DY%2E+Lin%2C+and+D%2E+Ravichandran%2E+2002b%2E+Using+Knowledge+to+Facilitate+Pinpointing+of+Factoid+Answers%2E+Proceedings+of+the+COLING%2D2002+conference%2E+Taipei%2C+Taiwan%2E) 7. Lee, G.G., J. Seo, S. Lee, H. Jung, B-H. Cho, C. Lee, B-K. Kwak, J, Cha, D. Kim, J-H. An, H. Kim, and K. Kim. 2001. SiteQ: Engineering High Performance QA System Using Lexico-Semantic Pattern Matching and Shallow NLP. Proceedings of the TREC-10 Conference. NIST, Gaithersburg, MD, 437–446. [find](https://scholar.google.com/scholar?q=Lee%2C+G%2EG%2E%2C+J%2E+Seo%2C+S%2E+Lee%2C+H%2E+Jung%2C+B%2DH%2E+Cho%2C+C%2E+Lee%2C+B%2DK%2E+Kwak%2C+J%2C+Cha%2C+D%2E+Kim%2C+J%2DH%2E+An%2C+H%2E+Kim%2C+and+K%2E+Kim%2E+2001%2E+SiteQ%3A+Engineering+High+Performance+QA+System+Using+Lexico%2DSemantic+Pattern+Matching) 8. Lin, C-Y. 2002. The Effectiveness of Dictionary and Web-Based Answer Reranking. Proceedings of the COLING-2002 conference. Taipei, Taiwan. [find](https://scholar.google.com/scholar?q=Lin%2C+C%2DY%2E+2002%2E+The+Effectiveness+of+Dictionary+and+Web%2DBased+Answer+Reranking%2E+Proceedings+of+the+COLING%2D2002+conference%2E+Taipei%2C+Taiwan%2E) 9. Prager, J. and J. Chu-Carroll. 2001. Use of WordNet Hypernyms for Answering What-Is Questions. Proceedings of the TREC-10 Conference. NIST, Gaithersburg, MD, 309– 316. [find](https://scholar.google.com/scholar?q=Prager%2C+J%2E+and+J%2E+Chu%2DCarroll%2E+2001%2E+Use+of+WordNet+Hypernyms+for+Answering+What%2DIs+Questions%2E+Proceedings+of+the+TREC%2D10+Conference%2E+NIST%2C+Gaithersburg%2C+MD%2C+309%E2%80%93+316%2E) 10. Riloff, E. 1996. Automatically Generating Extraction Patterns from Untagged Text. Proceedings of the Thirteenth National Conference on Artificial Intelligence (AAAI-96), 1044–1049. [find](https://scholar.google.com/scholar?q=Riloff%2C+E%2E+1996%2E+Automatically+Generating+Extraction+Patterns+from+Untagged+Text%2E+Proceedings+of+the+Thirteenth+National+Conference+on+Artificial+Intelligence+%28AAAI%2D96%29%2C+1044%E2%80%931049%2E) 11. Soubbotin, M.M. and S.M. Soubbotin. 2001. Patterns of Potential Answer Expressions as Clues to the Right Answer. Proceedings of the TREC-10 Conference. NIST, Gaithersburg, MD, 175–182. [find](https://scholar.google.com/scholar?q=Soubbotin%2C+M%2EM%2E+and+S%2EM%2E+Soubbotin%2E+2001%2E+Patterns+of+Potential+Answer+Expressions+as+Clues+to+the+Right+Answer%2E+Proceedings+of+the+TREC%2D10+Conference%2E+NIST%2C+Gaithersburg%2C+MD%2C+175%E2%80%93182%2E) 12. Srihari, R. and W. Li. 2000. A Question Answering System Supported by Information Extraction. Proceedings of the 1 st Meeting of the North American Chapter of the Association for Computational Linguistics (ANLP-NAACL-00), Seattle, WA, 166–172. [find](https://scholar.google.com/scholar?q=Srihari%2C+R%2E+and+W%2E+Li%2E+2000%2E+A+Question+Answering+System+Supported+by+Information+Extraction%2E+Proceedings+of+the+1+st+Meeting+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguis) 13. Voorhees, E. 2001. Overview of the Question Answering Track. Proceedings of the TREC-10 Conference. NIST, Gaithersburg, MD, 157– 165. [find](https://scholar.google.com/scholar?q=Voorhees%2C+E%2E+2001%2E+Overview+of+the+Question+Answering+Track%2E+Proceedings+of+the+TREC%2D10+Conference%2E+NIST%2C+Gaithersburg%2C+MD%2C+157%E2%80%93+165%2E) 14. Wang, B., H. Xu, Z. Yang, Y. Liu, X. Cheng, D. Bu, and S. Bai. 2001. TREC-10 Experiments at CAS-ICT: Filtering, Web, and QA. Proceedings of the TREC-10 Conference. NIST, Gaithersburg, MD, 229–241. [find](https://scholar.google.com/scholar?q=Wang%2C+B%2E%2C+H%2E+Xu%2C+Z%2E+Yang%2C+Y%2E+Liu%2C+X%2E+Cheng%2C+D%2E+Bu%2C+and+S%2E+Bai%2E+2001%2E+TREC%2D10+Experiments+at+CAS%2DICT%3A+Filtering%2C+Web%2C+and+QA%2E+Proceedings+of+the+TREC%2D10+Conference%2E+NIST%2C+Gaithersburg%2C+MD%2C+229%E2%80%93241%2E)