--- title: "The Value of Semantic Parse Labeling for Knowledge Base Question Answering — Yih 2016" description: "The paper examines whether it is worth collecting semantic parse annotations for knowledge base question answering… 3 references, cited by 254 papers here." authors: ["Wen-tau Yih", "Matthew Richardson", "Chris Meek", "Ming-Wei Chang", "Jina Suh"] year: 2016 venue: "Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/P16-2033/" doi: "10.18653/v1/p16-2033" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) / The Value of Semantic Parse Labeling for Knowledge Base Question Answering # The Value of Semantic Parse Labeling for Knowledge Base Question Answering **Authors:** [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/), [Matthew Richardson](https://md.research.elamur.ai/free/en/authors/matthew-richardson/), Chris Meek, [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/) and Jina Suh · **Year:** 2016 · **Venue:** Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P16-2033/) · [PDF](https://aclanthology.org/P16-2033.pdf) · [DOI 10.18653/v1/p16-2033](https://doi.org/10.18653/v1/p16-2033) **Citations:** [254](https://md.research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/cited-by/) in our library, of the 140,884 papers whose bibliographies we read ## Summary The paper examines whether it is worth collecting semantic parse annotations for knowledge base question answering (KBQA). Most current systems are trained only on question–answer pairs, and such annotation is usually considered expensive. The authors fully re-annotated WebQuestions on top of Freebase and, for every question the knowledge base is able to answer, wrote a SPARQL query that can be executed directly. To do this, they built a step-by-step, dialog-like interface. The annotator selects the topic entity, then the predicate leading to the answer, and finally adds refining constraints. The system itself assembles a consistent parse from these choices. This produced WebQuestionsSP, the largest dataset of its kind at the time. Annotating the parses took no more time than finding the answers themselves, and the quality was no lower. Training the STAGG system on parses instead of answers raised the average F1 by about 5 points. The authors argue that on large knowledge bases such annotation pays off: it filters out spurious paths between entities. In addition, the answers remain correct when the knowledge base is updated. ## Key points - **WebQuestionsSP** (http://aka.ms/WebQSP) contains SPARQL parses for 4,737 questions. Another 1,073 of 5,810 (about 18.5%) are marked as ones the knowledge base cannot answer: the questions are ambiguous or Freebase is not suited to them. The annotation was done on the latest version of Freebase (2015-08-09). - The annotation was carried out by five annotators who know Freebase well. Candidate entities were suggested by an entity linking system. About 3.1% of the queries were not covered by the interface, so they were finished by manual editing. - Training and evaluation: 3,098 questions for training, 1,639 for testing. With parses, the average F1 was 71.7% versus 66.8%, and the exact-match accuracy of the full answer set was 63.9% versus 58.8%. - Cost (a sample of 50 questions): an expert spent on average 21 s per question on a parse and 82 s on manually finding the answers. 94% of the parses and 92% of the manually collected answers turned out to be fully correct. In the original WebQuestions (MTurk), only 66% are. - Limitations: the interface is not designed for complex compositional questions. Where there are few possible paths between entities and answers, the gain from annotation may be small. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) Topics: [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/) ## Cite Cite as: Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang and Jina Suh. 2016. The Value of Semantic Parse Labeling for Knowledge Base Question Answering. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{yih2016value, title = {{The Value of Semantic Parse Labeling for Knowledge Base Question Answering}}, author = {Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang and Jina Suh}, booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2016}, doi = {10.18653/v1/p16-2033}, url = {https://doi.org/10.18653/v1/p16-2033} } ``` - [← previous Language to Logical Form with Neural Attention](https://md.research.elamur.ai/free/en/papers/dong-2016-language-logical-form-neural-attention/) - [next → Neural Semantic Role Labeling with Dependency Path Embeddings](https://md.research.elamur.ai/free/en/papers/roth-2016-neural-semantic-role-labeling-dependency/) - [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/page/2/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Hannah Bast and Elmar Haussmann. 2015. More accurate question answering on Freebase. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1431–1440. ACM. [find](https://scholar.google.com/scholar?q=Hannah+Bast+and+Elmar+Haussmann%2E+2015%2E+More+accurate+question+answering+on+Freebase%2E+In+Proceedings+of+the+24th+ACM+International+on+Conference+on+Information+and+Knowledge+Management%2C+pages+1431%E2%80%931440) 2. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425, Baltimore, Maryland, June. Association for Computational Linguistics. [our card](https://md.research.elamur.ai/free/en/papers/berant-2014-semantic-parsing-paraphrasing/) 3. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544, Seattle, Washington, USA, October. Association for Computational Ling… [our card](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/)