---
title: "The Web as a Knowledge-Base for Answering Complex Questions — Talmor 2018"
description: "The paper addresses complex open-domain questions that require combining several facts. 38.6 F1. 44 references, cited by 316 papers here."
authors: ["Alon Talmor", "Jonathan Berant"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)"
source: "https://aclanthology.org/N18-1059/"
doi: "10.18653/v1/n18-1059"
arxiv: "1803.06643"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/) / The Web as a Knowledge-Base for Answering Complex Questions

# The Web as a Knowledge-Base for Answering Complex Questions

**Authors:** [Alon Talmor](https://md.research.elamur.ai/free/en/authors/alon-talmor/) and [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-1059/) · [PDF](https://aclanthology.org/N18-1059.pdf) · [DOI 10.18653/v1/n18-1059](https://doi.org/10.18653/v1/n18-1059) · [arXiv 1803.06643](https://arxiv.org/abs/1803.06643)

**Citations:** [316](https://md.research.elamur.ai/free/en/papers/talmor-2018-web-as-knowledge-base-answering/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
The paper addresses complex open-domain questions that require combining several facts. The authors rely neither on a pre-built knowledge base nor on pre-supplied documents. A complex question is decomposed into a sequence of simpler ones, each of which is sent to a search engine. An RC model then finds the answer in web snippets, and the final answer is obtained through set operations such as union or intersection. To evaluate the approach, the ComplexWebQuestions dataset was created. It is based on SPARQL queries from WebQuestionsSP that were automatically made more complex by adding composition, conjunction, superlatives, and comparatives. Machine-generated questions were produced from these queries using templates, and Amazon Mechanical Turk workers rephrased them in natural language. The SplitQA model has a sequence-to-sequence architecture and maps a question to a short program, i.e., a computation tree. Its training data were obtained through noisy alignment of the machine-generated and human formulations. Decomposition raises precision@1 from 20.8 to 27.5, while a time-limited human reaches 63.0. The work shows that the web can replace a knowledge base for compositional questions. Decomposition helps primarily with retrieving relevant documents.

## Key points
- **ComplexWebQuestions** contains 34,689 question–answer pairs together with SPARQL queries and web snippets. For 462 KB predicates, 687 templates were manually annotated. The dataset is available at nlp.cs.tau.ac.il/compwebq, and the code is in the WebAsKB repository on GitHub.
- **The formal language** consists of the functions SimpQA, Comp (substitutes answers for VAR), Conj (set intersection), and Add. Superlatives, comparatives, and negation are not decomposed but are passed to SimpQA directly. In the paper's running example, the answer is Lviv.
- **Results on ComplexWebQuestions (p@1):**
  - SimpQA without decomposition — 20.8, SplitQA — 27.5, oracle — 33.7.
  - Another RC model also benefits from decomposition: SplitRCQA outperforms RCQA by 3.4 points.
  - The Google box gives the correct answer in only 2.5% of cases.
  - A human with a limit of 4 minutes per question reaches 63.0.
- **ComplexQuestions** (Bao et al., 2016) is a small dataset for semantic parsing over Freebase:
  - SimpQA obtains 38.6 F1, and a simple rule that splits on the words "when/during" (SplitQARule) obtains 39.7 F1.
  - The best system, CompQ, which works directly with Freebase, achieves 40.9 F1.
  - On dev, SplitQARule yields 43.1 F1. After manually fixing string matches (SplitQARule++), the result rises to 46.9.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · [🕸️ Knowledge graphs & ontologies](https://md.research.elamur.ai/free/en/topics/knowledge-graphs/)

## Cite

Cite as: Alon Talmor and Jonathan Berant. 2018. The Web as a Knowledge-Base for Answering Complex Questions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{talmor2018web,
  title = {{The Web as a Knowledge-Base for Answering Complex Questions}},
  author = {Alon Talmor and Jonathan Berant},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1059},
  eprint = {1803.06643},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-1059}
}
```

- [← previous CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies](https://md.research.elamur.ai/free/en/papers/zeman-2018-conll-2018-shared-task-multilingual/)
- [next → Overview of the Evalita 2018 Task on Automatic Misogyny Identification (AMI)](https://md.research.elamur.ai/free/en/papers/fersini-2018-overview-evalita-2018-task-automatic/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/page/2/)

## References

44 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. K. M. Hermann, T. Koisk, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS). [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
2. Y. Artzi and L. Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. Transactions of the Association for Computational Linguistics (TACL) 1:49–62. [our card](https://md.research.elamur.ai/free/en/papers/artzi-2013-weakly-supervised-learning-semantic-parsers/)
3. D. Hewlett, A. Lacoste, L. Jones, I. Polosukhin, A. Fandrianto, J. Han, M. Kelcey, and D. Berthelot. 2016. Wikireading: A novel large-scale language understanding task over Wikipedia. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=D%2E+Hewlett%2C+A%2E+Lacoste%2C+L%2E+Jones%2C+I%2E+Polosukhin%2C+A%2E+Fandrianto%2C+J%2E+Han%2C+M%2E+Kelcey%2C+and+D%2E+Berthelot%2E+2016%2E+Wikireading%3A+A+novel+large%2Dscale+language+understanding+task+over+Wikipedia%2E+In+Association+f)
4. J. Bao, N. Duan, Z. Yan, M. Zhou, and T. Zhao. 2016. Constraint-based question answering with knowledge graph. In International Conference on Computational Linguistics (COLING). [find](https://scholar.google.com/scholar?q=J%2E+Bao%2C+N%2E+Duan%2C+Z%2E+Yan%2C+M%2E+Zhou%2C+and+T%2E+Zhao%2E+2016%2E+Constraint%2Dbased+question+answering+with+knowledge+graph%2E+In+International+Conference+on+Computational+Linguistics+%28COLING%29%2E)
5. J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In Empirical Methods in Natural Language Processing (EMNLP). [our card](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/)
6. F. Hill, A. Bordes, S. Chopra, and J. Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR). [our card](https://md.research.elamur.ai/free/en/papers/hill-2015-goldilocks-principle-reading-children-s/)
7. J. Berant and P. Liang. 2015. Imitation learning of agenda-based semantic parsers. Transactions of the Association for Computational Linguistics (TACL) 3:545–558. [find](https://scholar.google.com/scholar?q=J%2E+Berant+and+P%2E+Liang%2E+2015%2E+Imitation+learning+of+agenda%2Dbased+semantic+parsers%2E+Transactions+of+the+Association+for+Computational+Linguistics+%28TACL%29+3%3A545%E2%80%93558%2E)
8. M. Iyyer, W. Yih, and M. Chang. 2016. Answering complicated question intents expressed in decomposed question sequences. CoRR 0. [find](https://scholar.google.com/scholar?q=M%2E+Iyyer%2C+W%2E+Yih%2C+and+M%2E+Chang%2E+2016%2E+Answering+complicated+question+intents+expressed+in+decomposed+question+sequences%2E+CoRR+0%2E)
9. K. Bollacker, C. Evans, P. Paritosh, T. Sturge, and J. Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In International Conference on Management of Data (SIGMOD). pages 1247–1250. [find](https://scholar.google.com/scholar?q=K%2E+Bollacker%2C+C%2E+Evans%2C+P%2E+Paritosh%2C+T%2E+Sturge%2C+and+J%2E+Taylor%2E+2008%2E+Freebase%3A+a+collaboratively+created+graph+database+for+structuring+human+knowledge%2E+In+International+Conference+on+Management+of+Da)
10. R. Jia and P. Liang. 2016. Data recombination for neural semantic parsing. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=R%2E+Jia+and+P%2E+Liang%2E+2016%2E+Data+recombination+for+neural+semantic+parsing%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E)
11. R. Jia and P. Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Empirical Methods in Natural Language Processing (EMNLP). [our card](https://md.research.elamur.ai/free/en/papers/jia-2017-adversarial-examples-evaluating-reading-comprehension/)
12. C. Buck, J. Bulian, M. Ciaramita, A. Gesmundo, N. Houlsby, W. Gajewski, and W. Wang. 2017. Ask the right questions: Active question reformulation with reinforcement learning. arXiv preprint arXiv:1705.07830 . — [arXiv:1705.07830](https://arxiv.org/abs/1705.07830) [find](https://scholar.google.com/scholar?q=C%2E+Buck%2C+J%2E+Bulian%2C+M%2E+Ciaramita%2C+A%2E+Gesmundo%2C+N%2E+Houlsby%2C+W%2E+Gajewski%2C+and+W%2E+Wang%2E+2017%2E+Ask+the+right+questions%3A+Active+question+reformulation+with+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1705)
13. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL). [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
14. D. Chen, J. Bolton, and C. D. Manning. 2016. A thorough examination of the CNN / Daily Mail reading comprehension task. In Association for Computational Linguistics (ACL). [our card](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/)
15. D. Chen, A. Fisch, J. Weston, and A. Bordes. 2017. Reading Wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051 . [our card](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/)
16. J. Krishnamurthy and T. Mitchell. 2012. Weakly supervised training of semantic parsers. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP/CoNLL). pages 754–765. [find](https://scholar.google.com/scholar?q=J%2E+Krishnamurthy+and+T%2E+Mitchell%2E+2012%2E+Weakly+supervised+training+of+semantic+parsers%2E+In+Empirical+Methods+in+Natural+Language+Processing+and+Computational+Natural+Language+Learning+%28EMNLP%2FCoNLL%29%2E+p)
17. K. Cho, B. van Merriënboer, D. Bahdanau, and Y. Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 . [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
18. T. Kwiatkowski, E. Choi, Y. Artzi, and L. Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Empirical Methods in Natural Language Processing (EMNLP). [our card](https://md.research.elamur.ai/free/en/papers/kwiatkowski-2013-scaling-semantic-parsers-fly-ontology/)
19. C. Clark and M. Gardner. 2017. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723 . [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)
20. G. Lai, Q. Xie, H. Liu, Y. Yang, and E. Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683 . [our card](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/)
21. J. Duchi, E. Hazan, and Y. Singer. 2010. Adaptive subgradient methods for online learning and stochastic optimization. In Conference on Learning Theory (COLT). [find](https://scholar.google.com/scholar?q=J%2E+Duchi%2C+E%2E+Hazan%2C+and+Y%2E+Singer%2E+2010%2E+Adaptive+subgradient+methods+for+online+learning+and+stochastic+optimization%2E+In+Conference+on+Learning+Theory+%28COLT%29%2E)
22. K. lee, M. Lewis, and L. Zettlemoyer. 2016. Global neural CCG parsing with optimality guarantees. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=K%2E+lee%2C+M%2E+Lewis%2C+and+L%2E+Zettlemoyer%2E+2016%2E+Global+neural+CCG+parsing+with+optimality+guarantees%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
23. M. Dunn, , L. Sagun, M. Higgins, U. Guney, V. Cirik, and K. Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv . [our card](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/)
24. P. Liang. 2013. Lambda dependency-based compositional semantics. arXiv preprint arXiv:1309.4408 . [find](https://scholar.google.com/scholar?q=P%2E+Liang%2E+2013%2E+Lambda+dependency%2Dbased+compositional+semantics%2E+arXiv+preprint+arXiv%3A1309%2E4408+%2E)
25. P. Liang, M. I. Jordan, and D. Klein. 2011. Learning dependency-based compositional semantics. In Association for Computational Linguistics (ACL). pages 590–599. [find](https://scholar.google.com/scholar?q=P%2E+Liang%2C+M%2E+I%2E+Jordan%2C+and+D%2E+Klein%2E+2011%2E+Learning+dependency%2Dbased+compositional+semantics%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E+pages+590%E2%80%93599%2E)
26. K. Guu, P. Pasupat, E. Z. Liu, and P. Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Association for Computational Linguistics (ACL). [our card](https://md.research.elamur.ai/free/en/papers/guu-2017-language-programs-bridging-reinforcement-learning/)
27. W. Yih, M. Richardson, C. Meek, M. Chang, and J. Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Association for Computational Linguistics (ACL). [our card](https://md.research.elamur.ai/free/en/papers/yih-2016-value-semantic-parse-labeling-knowledge/)
28. T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Workshop on Cognitive Computing at NIPS. [our card](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/)
29. M. Zelle and R. J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Association for the Advancement of Artificial Intelligence (AAAI). pages 1050–1055. [our card](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/)
30. R. Nogueira and K. Cho. 2016. End-to-end goal-driven web navigation. In Advances in Neural Information Processing Systems (NIPS). [find](https://scholar.google.com/scholar?q=R%2E+Nogueira+and+K%2E+Cho%2E+2016%2E+End%2Dto%2Dend+goal%2Ddriven+web+navigation%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2E)
31. T. Onishi, H. Wang, M. Bansal, K. Gimpel, and D. McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Empirical Methods in Natural Language Processing (EMNLP). [find](https://scholar.google.com/scholar?q=T%2E+Onishi%2C+H%2E+Wang%2C+M%2E+Bansal%2C+K%2E+Gimpel%2C+and+D%2E+McAllester%2E+2016%2E+Who+did+what%3A+A+large%2Dscale+person%2Dcentered+cloze+dataset%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2E)
32. L. S. Zettlemoyer and M. Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence (UAI). pages 658– 666. [our card](https://md.research.elamur.ai/free/en/papers/zettlemoyer-2012-learning-map-sentences-logical-form/)
33. P. Pasupat and P. Liang. 2015. Compositional semantic parsing on semi-structured tables. In Association for Computational Linguistics (ACL). [our card](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/)
34. V. Zhong, C. Xiong, and R. Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103 . [find](https://scholar.google.com/scholar?q=V%2E+Zhong%2C+C%2E+Xiong%2C+and+R%2E+Socher%2E+2017%2E+Seq2sql%3A+Generating+structured+queries+from+natural+language+using+reinforcement+learning%2E+arXiv+preprint+arXiv%3A1709%2E00103+%2E)
35. J. Pennington, R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
36. P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
37. The Syntactic Process. MIT I. Sutskever, O. Vinyals, and Q. V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS). pages 3104–3112. [find](https://scholar.google.com/scholar?q=The+Syntactic+Process%2E+MIT+I%2E+Sutskever%2C+O%2E+Vinyals%2C+and+Q%2E+V%2E+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2E+pages+3104%E2%80%93311)
38. A. Talmor, M. Geva, and J. Berant. 2017. Evaluating semantic parsing against a simple web-based question answering model. In *SEM. [find](https://scholar.google.com/scholar?q=A%2E+Talmor%2C+M%2E+Geva%2C+and+J%2E+Berant%2E+2017%2E+Evaluating+semantic+parsing+against+a+simple+web%2Dbased+question+answering+model%2E+In+%2ASEM%2E)
39. O. Vinyals, M. Fortunato, and N. Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems (NIPS). pages 2674–2682. [find](https://scholar.google.com/scholar?q=O%2E+Vinyals%2C+M%2E+Fortunato%2C+and+N%2E+Jaitly%2E+2015%2E+Pointer+networks%2E+In+Advances+in+Neural+Information+Processing+Systems+%28NIPS%29%2E+pages+2674%E2%80%932682%2E)
40. W. Wang, N. Yang, F. Wei, B. Chang, and M. Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Association for Computational Linguistics (ACL). [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)
41. Y. Wang, J. Berant, and P. Liang. 2015. Building a semantic parser overnight. In Association for Computational Linguistics (ACL). [find](https://scholar.google.com/scholar?q=Y%2E+Wang%2C+J%2E+Berant%2C+and+P%2E+Liang%2E+2015%2E+Building+a+semantic+parser+overnight%2E+In+Association+for+Computational+Linguistics+%28ACL%29%2E)
42. Y. Watanabe, B. Dhingra, and R. Salakhutdinov. 2017. Question answering from unstructured text by retrieval and comprehension. arXiv preprint arXiv:1703.08885 . — [arXiv:1703.08885](https://arxiv.org/abs/1703.08885) [find](https://scholar.google.com/scholar?q=Y%2E+Watanabe%2C+B%2E+Dhingra%2C+and+R%2E+Salakhutdinov%2E+2017%2E+Question+answering+from+unstructured+text+by+retrieval+and+comprehension%2E+arXiv+preprint+arXiv%3A1703%2E08885+%2E+%E2%80%94+%5BarXiv%3A1703%2E08885%5D%28https%3A%2F%2Farxiv%2Eorg%2F)
43. J. Welbl, P. Stenetorp, and S. Riedel. 2017. Constructing datasets for multi-hop reading comprehension across documents. arXiv preprint arXiv:1710.06481 . [our card](https://md.research.elamur.ai/free/en/papers/welbl-2018-constructing-datasets-multi-hop-reading/)
44. Y. Yang, W. Yih, and C. Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP). pages 2013–2018. [our card](https://md.research.elamur.ai/free/en/papers/yang-2015-wikiqa-challenge-dataset-open-domain/)