---
title: "Querying Knowledge via Multi-Hop English Questions — Gao 2019"
description: "The paper describes how the frame-semantic parser is built incrementally through… 100% accuracy. 15 references, cited by 1 paper here."
authors: ["Tiantian Gao", "Paul Fodor", "Michael Kifer"]
year: 2019
venue: "arXiv:1907.08176 · Theory and Practice of Logic Programming 19 (2019) 636-653"
source: "https://arxiv.org/abs/1907.08176"
doi: "10.1017/s1471068419000103"
arxiv: "1907.08176"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gao-2019-querying-knowledge-multi-hop-english/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gao-2019-querying-knowledge-multi-hop-english/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/) / Querying Knowledge via Multi-Hop English Questions

# Querying Knowledge via Multi-Hop English Questions

**Authors:** Tiantian Gao, [Paul Fodor](https://md.research.elamur.ai/free/en/authors/paul-fodor/) and [Michael Kifer](https://md.research.elamur.ai/free/en/authors/michael-kifer/) · **Year:** 2019 · **Venue:** arXiv:1907.08176 · Theory and Practice of Logic Programming 19 (2019) 636-653 · **Publisher:** [Cambridge University Press](https://md.research.elamur.ai/free/en/publishers/cambridge-university-press/) · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://doi.org/10.1017/s1471068419000103) · [PDF](https://arxiv.org/pdf/1907.08176) · [DOI 10.1017/s1471068419000103](https://doi.org/10.1017/s1471068419000103) · [arXiv 1907.08176](https://arxiv.org/abs/1907.08176) · [OpenAlex](https://openalex.org/W2956661464)

**Citations:** 7 citations (OpenAlex, 2026-10-14) · [1](https://md.research.elamur.ai/free/en/papers/gao-2019-querying-knowledge-multi-hop-english/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
KALM-QA extends the KALM knowledge-authoring framework so that it can answer complex multi-hop English questions over a knowledge base. A multi-hop question requires joining several relations, possibly a relation with itself. Plain controlled natural languages such as ACE give different logical forms to sentences that mean the same thing. KALM fixes this with a semantic layer on top of ACE's parser, APE. The parser's DRS output is mapped onto FrameOnt, a formalized version of FrameNet frames. Role fillers are disambiguated against BabelNet synsets, so that equivalent sentences receive one unique logical representation (ULR), which is then run as a Prolog query. The paper describes how the frame-semantic parser is built incrementally through structure learning. MetaQA questions are rewritten with the Stanford Parser so that they fit ACE syntax. On MetaQA's movie questions (about 29,000 test, over 260,000 training), KALM-QA answers 100% correctly. The machine-learning system VRN scores far lower. The authors also show that many of MetaQA's own gold answers are wrong. The work argues that logic-based pipelines with explicit linguistic knowledge can reach the precision that knowledge representation needs.

## Key points
- The pipeline runs in this order: APE syntactic parse to DRS, frame-based parsing with logical valence patterns (lvps), role-filler disambiguation over BabelNet, then construction of logic facts or queries.
- On MetaQA's 2-hop and 3-hop questions, KALM-QA reaches 100% accuracy. VRN (Zhang et al. 2018b) falls well short.
- For MetaQA, KALM-QA had to learn 88 lvps. The authors estimate that capturing the same semantics in bare Attempto or SBVR would take 974 hand-written background bridge rules.
- MetaQA mislabels a large share of answers, mainly because distinct movies can share a title. As a result, end-to-end learning learned wrong answers for more than half of the 3-hop queries.
- Validation used standardized query templates (44 + 59, with several English variants each) and a few hundred manual checks. An error report is published at github.com/tiantiangao7/kalm-qa.

## Interesting ideas
- An lvp ties a trigger word to a frame and gives each role its own small grammatical-path extraction rule, such as verb->subject; a pattern applies only if every required role can be filled.
- Role fillers are stored as synsets rather than words, and some roles carry type constraints, such as the Release Year role requiring a valid year.
- Wrong frame readings are pruned by checking whether a filler can fit the role's type: "directs a film" is not about running a corporation, because a film is not a corporation.
- To verify tens of thousands of translations cheaply, constants are masked (for example, 'Bright Star' becomes xxxx), outputs are grouped by template, and only one representative per template is checked by hand.
- The authors audited the benchmark's gold answers with a script that explains each mismatch, such as distinct films sharing a title being conflated.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): Frames could be triggered from UD trees as lemma plus dependency-path rules, one per VerbNet role, so every extracted role traces back to an explicit path.
- **I2** (world): Entities and states in the world model could be keyed to WordNet/Wikidata senses, with typed slots that reject values that cannot fit.
- **I3** (frames): Ambiguous verb frames could be resolved by checking argument compatibility against the ontology, and the failed check could be logged as the explanation for the choice.
- **I4** (learn): The same trick could audit silver frame or query outputs at scale by reviewing per-template exemplars instead of every sentence.
- **I5** (answer): QA on tales and school tests should keep entity IDs separate from surface names, and gold labels should be checked automatically before disagreements are counted as Mova errors.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stony Brook University](https://md.research.elamur.ai/free/en/universities/stony-brook-university/) — University, Stony Brook, United States · 27 works · 2,057 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [English](https://md.research.elamur.ai/free/en/topics/english/) · 1,373 works

## Cite

Cite as: Tiantian Gao, Paul Fodor and Michael Kifer. 2019. Querying Knowledge via Multi-Hop English Questions. arXiv:1907.08176 · Theory and Practice of Logic Programming 19 (2019) 636-653.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{gao2019querying,
  title = {{Querying Knowledge via Multi-Hop English Questions}},
  author = {Tiantian Gao, Paul Fodor and Michael Kifer},
  year = {2019},
  doi = {10.1017/s1471068419000103},
  eprint = {1907.08176},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1017/s1471068419000103}
}
```

- [← previous Proceedings of the Third Workshop on Universal Dependencies (UDW, SyntaxFest 2019)](https://md.research.elamur.ai/free/en/papers/anon-2019-proceedings-third-workshop-universal-dependencies/)
- [next → Rethinking templates: A syntactic analysis of verbal morphology in Emirati Arabic](https://md.research.elamur.ai/free/en/papers/kaabi-2019-rethinking-templates-syntactic-analysis-verbal/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/)

## References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Tiantian Gao, Paul Fodor, Michael Kifer J OHNSON , C. R., F ILLMORE , C. J., P ETRUCK , M. R., B AKER , C. F., E LLSWORTH , M. J., R UPPEN - HOFER , J., AND W OOD , E. J. 2002. FrameNet: Theory and Practice. [find](https://scholar.google.com/scholar?q=Tiantian+Gao%2C+Paul+Fodor%2C+Michael+Kifer+J+OHNSON+%2C+C%2E+R%2E%2C+F+ILLMORE+%2C+C%2E+J%2E%2C+P+ETRUCK+%2C+M%2E+R%2E%2C+B+AKER+%2C+C%2E+F%2E%2C+E+LLSWORTH+%2C+M%2E+J%2E%2C+R+UPPEN+%2D+HOFER+%2C+J%2E%2C+AND+W+OOD+%2C+E%2E+J%2E+2002%2E+FrameNet%3A+Theory+and+Pr)
2. K UHN , T. 2014. A survey and classification of controlled natural languages. Comp. Linguistics 40, 1, 121–170. [find](https://scholar.google.com/scholar?q=K+UHN+%2C+T%2E+2014%2E+A+survey+and+classification+of+controlled+natural+languages%2E+Comp%2E+Linguistics+40%2C+1%2C+121%E2%80%93170%2E)
3. L I , F. AND J AGADISH , H. V. 2014. Constructing an interactive natural language interface for relational databases. PVLDB 8, 1, 73–84. [find](https://scholar.google.com/scholar?q=L+I+%2C+F%2E+AND+J+AGADISH+%2C+H%2E+V%2E+2014%2E+Constructing+an+interactive+natural+language+interface+for+relational+databases%2E+PVLDB+8%2C+1%2C+73%E2%80%9384%2E)
4. L ÓPEZ , V., F ERN ÁNDEZ , M., M OTTA , E., AND S TIELER , N. 2012. PowerAqua: Supporting users in querying and exploring the semantic web. Semantic Web 3, 3, 249–265. [find](https://scholar.google.com/scholar?q=L+O%CC%81PEZ+%2C+V%2E%2C+F+ERN+A%CC%81NDEZ+%2C+M%2E%2C+M+OTTA+%2C+E%2E%2C+AND+S+TIELER+%2C+N%2E+2012%2E+PowerAqua%3A+Supporting+users+in+querying+and+exploring+the+semantic+web%2E+Semantic+Web+3%2C+3%2C+249%E2%80%93265%2E)
5. M ANNING , C. D., S URDEANU , M., B AUER , J., F INKEL , J., B ETHARD , S. J., AND M C C LOSKY , D. 2014. [The Stanford CoreNLP natural language processing toolkit](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/).
6. M AUSAM , S CHMITZ , M., S ODERLAND , S., B ART , R., AND E TZIONI , O. 2012. Open language learning for information extraction. In The Joint Conf. on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, EMNLP-CoNLL. Association for Computational Linguistics, Jeju Island, Korea, 523–534. [find](https://scholar.google.com/scholar?q=M+AUSAM+%2C+S+CHMITZ+%2C+M%2E%2C+S+ODERLAND+%2C+S%2E%2C+B+ART+%2C+R%2E%2C+AND+E+TZIONI+%2C+O%2E+2012%2E+Open+language+learning+for+information+extraction%2E+In+The+Joint+Conf%2E+on+Empirical+Methods+in+Natural+Language+Processing+)
7. M C I LRAITH , S. A. AND W EINBERGER , K. Q., Eds. 2018. 32nd AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. AAAI Press. [find](https://scholar.google.com/scholar?q=M+C+I+LRAITH+%2C+S%2E+A%2E+AND+W+EINBERGER+%2C+K%2E+Q%2E%2C+Eds%2E+2018%2E+32nd+AAAI+Conference+on+Artificial+Intelligence%2C+%28AAAI%2D18%29%2C+the+30th+innovative+Applications+of+Artificial+Intelligence+%28IAAI%2D18%29%2C+and+the+8th+)
8. M ILLER , A. H., F ISCH , A., D ODGE , J., K ARIMI , A., B ORDES , A., AND W ESTON , J. 2016. Key-value memory networks for directly reading documents. In 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP), J. Su, X. Carreras, and K. Duh, Eds. The Association for Computational Linguistics, Austin, TX, 1400–1409. [find](https://scholar.google.com/scholar?q=M+ILLER+%2C+A%2E+H%2E%2C+F+ISCH+%2C+A%2E%2C+D+ODGE+%2C+J%2E%2C+K+ARIMI+%2C+A%2E%2C+B+ORDES+%2C+A%2E%2C+AND+W+ESTON+%2C+J%2E+2016%2E+Key%2Dvalue+memory+networks+for+directly+reading+documents%2E+In+2016+Conference+on+Empirical+Methods+in+Natur)
9. N AVIGLI , R. AND P ONZETTO , S. P. 2012. BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network. Artificial Intelligence 193, 217–250. [find](https://scholar.google.com/scholar?q=N+AVIGLI+%2C+R%2E+AND+P+ONZETTO+%2C+S%2E+P%2E+2012%2E+BabelNet%3A+The+automatic+construction%2C+evaluation+and+application+of+a+wide%2Dcoverage+multilingual+semantic+network%2E+Artificial+Intelligence+193%2C+217%E2%80%93250%2E)
10. O BJECT -M ANAGEMENT -G ROUP . 2017. Semantics of business vocabulary and business rules (SBVR), v. 1.4. OMG standards document. [http://www.omg.org/spec/SBVR/Current](http://www.omg.org/spec/SBVR/Current). [↗](http://www.omg.org/spec/SBVR/Current)
11. R INGGAARD , M., G UPTA , R., AND P EREIRA , F. C. N. 2017. SLING: A framework for frame semantic parsing. CoRR 1710.07032, 1–9. [find](https://scholar.google.com/scholar?q=R+INGGAARD+%2C+M%2E%2C+G+UPTA+%2C+R%2E%2C+AND+P+EREIRA+%2C+F%2E+C%2E+N%2E+2017%2E+SLING%3A+A+framework+for+frame+semantic+parsing%2E+CoRR+1710%2E07032%2C+1%E2%80%939%2E)
12. S AHA , D., F LORATOU , A., S ANKARANARAYANAN , K., M INHAS , U. F., M ITTAL , A. R., AND Ö ZCAN , F. 2016. ATHENA: an ontology-driven system for natural language querying over relational data stores. PVLDB 9, 12, 1209–1220. [find](https://scholar.google.com/scholar?q=S+AHA+%2C+D%2E%2C+F+LORATOU+%2C+A%2E%2C+S+ANKARANARAYANAN+%2C+K%2E%2C+M+INHAS+%2C+U%2E+F%2E%2C+M+ITTAL+%2C+A%2E+R%2E%2C+AND+O%CC%88+ZCAN+%2C+F%2E+2016%2E+ATHENA%3A+an+ontology%2Ddriven+system+for+natural+language+querying+over+relational+data+stores)
13. S CHWITTER , R. 2010. Controlled natural languages for knowledge representation. In COLING 2010, 23rd Intl. Conf. on Computational Linguistics, Posters Volume, 23-27. ACL, Beijing, China, 1113–1121. [find](https://scholar.google.com/scholar?q=S+CHWITTER+%2C+R%2E+2010%2E+Controlled+natural+languages+for+knowledge+representation%2E+In+COLING+2010%2C+23rd+Intl%2E+Conf%2E+on+Computational+Linguistics%2C+Posters+Volume%2C+23%2D27%2E+ACL%2C+Beijing%2C+China%2C+1113%E2%80%931121%2E)
14. Z HANG , Y., D AI , H., K OZAREVA , Z., S MOLA , A. J., AND S ONG , L. 2018a. The MetaQA dataset. [https://github.com/yuyuz/MetaQA](https://github.com/yuyuz/MetaQA). [↗](https://github.com/yuyuz/MetaQA)
15. Z HANG , Y., D AI , H., K OZAREVA , Z., S MOLA , A. J., AND S ONG , L. 2018b. Variational reasoning for question answering with knowledge graph. See McIlraith and Weinberger (2018), 6069–6076. [Querying Knowledge via Multi-Hop English Questions](https://md.research.elamur.ai/free/en/papers/gao-2019-querying-knowledge-multi-hop-english/)

*Updated 24 October 2026*