---
title: "QuaRTz — paper by Tafjord et al. (2019) — summary and ideas"
description: "Answering questions that depend on qualitative relationships (for example, more of X leads to less of… 15 references, cited by 80 papers here."
authors: ["Oyvind Tafjord", "Matt Gardner", "Kevin Lin", "Peter Clark"]
year: 2019
venue: "arXiv:1909.03553"
source: "https://arxiv.org/abs/1909.03553"
doi: "10.48550/arxiv.1909.03553"
arxiv: "1909.03553"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/tafjord-2019-quartz-open-domain-dataset-qualitative/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tafjord-2019-quartz-open-domain-dataset-qualitative/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/) / QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

# QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions

**Authors:** [Oyvind Tafjord](https://md.research.elamur.ai/free/en/authors/oyvind-tafjord/), [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/), [Kevin Lin](https://md.research.elamur.ai/free/en/authors/kevin-lin/) and [Peter Clark](https://md.research.elamur.ai/free/en/authors/peter-clark/) · **Year:** 2019 · **Venue:** arXiv:1909.03553 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1909.03553) · [PDF](https://arxiv.org/pdf/1909.03553) · [DOI 10.48550/arxiv.1909.03553](https://doi.org/10.48550/arxiv.1909.03553) · [arXiv 1909.03553](https://arxiv.org/abs/1909.03553) · [OpenAlex](https://openalex.org/W2971613471)

**Citations:** 1 citations (OpenAlex, 2026-10-14) · [80](https://md.research.elamur.ai/free/en/papers/tafjord-2019-quartz-open-domain-dataset-qualitative/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Answering questions that depend on qualitative relationships (for example, more of X leads to less of Y) is hard for QA systems. The only earlier dataset focused on this, QuaRel, encoded its knowledge in a small formal ontology. QuaRTz is the first open-domain alternative, and its background knowledge is plain text. The authors used keyword search to pick 400 general qualitative statements from a large corpus. Crowdworkers then wrote situated two-choice questions grounded in each statement and annotated the properties being compared. A second group of workers checked each question and wrote a "flipped" version with the opposite answer, so that systems cannot succeed by simply matching comparative words. The task is to answer each question using the 400-sentence corpus. The strongest BERT-large model is first trained on RACE, then fine-tuned with retrieved knowledge. It reaches 73.7%, while humans score 95%. This leaves a gap of about 20 points and shows that applying arbitrary textual qualitative knowledge is still an open problem.

## Key points
- **Dataset size and splits:** QuaRTz has 3864 two-way multiple-choice questions, split 2696/384/784 for train/dev/test, and a background corpus of 400 knowledge sentences. Questions from the same seed sentence stay in the same split.
- **Flipped questions:** Half of the questions (1932) are flipped versions. When crowdworkers wrote freely, they flipped the relationship only about 10% of the time, and training on that data alone gave below-random accuracy on flipped questions.
- **Accuracy results:**
  - BERT-Sci: 54.6
  - BERT (IR), no RACE pre-fine-tuning: 64.4
  - BERT-PFT with no knowledge: 68.8
  - BERT-PFT (IR): 73.7
  - BERT-PFT with ideal knowledge (IR upper bound): 79.8
  - Humans: 95.0
- **Analysis of 100 sampled questions:** 68% use comparative wording that differs from the knowledge sentence, about 35% need commonsense, and 26% compare two explicit situations ("worlds").
- **Annotations and implementation:** The property annotations can support semantic-parsing approaches. All models were built with AllenNLP, and the data is available at data.allenai.org/quartz.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 304 works · 106,137 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

## Cite

Cite as: Oyvind Tafjord, Matt Gardner, Kevin Lin and Peter Clark. 2019. QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions. arXiv:1909.03553.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{tafjord2019quartz,
  title = {{QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions}},
  author = {Oyvind Tafjord, Matt Gardner, Kevin Lin and Peter Clark},
  year = {2019},
  doi = {10.48550/arxiv.1909.03553},
  eprint = {1909.03553},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1909.03553}
}
```

- [← previous On the Complexity and Typology of Inflectional Morphological Systems](https://md.research.elamur.ai/free/en/papers/cotterell-2019-complexity-typology-inflectional-morphological-systems/)
- [next → HOList: An Environment for Machine Learning of Higher-Order Theorem Proving](https://md.research.elamur.ai/free/en/papers/bansal-2019-holist-environment-machine-learning-higher/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/6/)

## References

15 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In EMNLP’13.
2. Christopher Clark and Matt Gardner. 2018. [Simple and effective multi-paragraph reading comprehension](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/). In ACL.
3. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. [Think you have solved question answering? Try ARC, the AI2 reasoning challenge](https://md.research.elamur.ai/free/en/papers/clark-2018-think-have-solved-question-answering/). arXiv preprint arXiv:1803.05457.
4. M. Crouse, C. McFate, and K. Forbus. 2018. Learning to build qualitative scenario models from natural language. In Proc. 31st Int. Workshop on Qualitative Reasoning (QR’18). [find](https://scholar.google.com/scholar?q=M%2E+Crouse%2C+C%2E+McFate%2C+and+K%2E+Forbus%2E+2018%2E+Learning+to+build+qualitative+scenario+models+from+natural+language%2E+In+Proc%2E+31st+Int%2E+Workshop+on+Qualitative+Reasoning+%28QR%E2%80%9918%29%2E)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
6. Kenneth D. Forbus. 1984. Qualitative process theory. Artificial Intelligence, 24:85–168. [find](https://scholar.google.com/scholar?q=Kenneth+D%2E+Forbus%2E+1984%2E+Qualitative+process+theory%2E+Artificial+Intelligence%2C+24%3A85%E2%80%93168%2E)
7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. [AllenNLP: A deep semantic natural language processing platform](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/). In NLP OSS Workshop at ACL. (arXiv:1803.07640).
8. Jayant Krishnamurthy, Pradeep Dasigi, and Matthew Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In EMNLP’17. [find](https://scholar.google.com/scholar?q=Jayant+Krishnamurthy%2C+Pradeep+Dasigi%2C+and+Matthew+Gardner%2E+2017%2E+Neural+semantic+parsing+with+type+constraints+for+semi%2Dstructured+tables%2E+In+EMNLP%E2%80%9917%2E)
9. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. [Race: Large-scale reading comprehension dataset from examinations](https://md.research.elamur.ai/free/en/papers/lai-2017-race-large-scale-reading-comprehension/). In EMNLP.
10. Clifton McFate and Kenneth Forbus. 2016. Scaling up linguistic processing of qualitative processes. In Proc. 4th Ann. Conf. on Advances in Cognitive Systems. [find](https://scholar.google.com/scholar?q=Clifton+McFate+and+Kenneth+Forbus%2E+2016%2E+Scaling+up+linguistic+processing+of+qualitative+processes%2E+In+Proc%2E+4th+Ann%2E+Conf%2E+on+Advances+in+Cognitive+Systems%2E)
11. Clifton James McFate, Kenneth D Forbus, and Thomas R Hinrichs. 2014. Using narrative function to extract qualitative information from natural language texts. In AAAI’14. [find](https://scholar.google.com/scholar?q=Clifton+James+McFate%2C+Kenneth+D+Forbus%2C+and+Thomas+R+Hinrichs%2E+2014%2E+Using+narrative+function+to+extract+qualitative+information+from+natural+language+texts%2E+In+AAAI%E2%80%9914%2E)
12. Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2019. [Improving machine reading comprehension with general reading strategies](https://md.research.elamur.ai/free/en/papers/sun-2019-improving-machine-reading-comprehension-general/). In NAACL.
13. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. Quarel: A dataset and models for answering questions about qualitative relationships. In AAAI. [find](https://scholar.google.com/scholar?q=Oyvind+Tafjord%2C+Peter+Clark%2C+Matt+Gardner%2C+Wen%2Dtau+Yih%2C+and+Ashish+Sabharwal%2E+2019%2E+Quarel%3A+A+dataset+and+models+for+answering+questions+about+qualitative+relationships%2E+In+AAAI%2E)
14. Daniel S Weld and Johan De Kleer. 2013. Readings in qualitative reasoning about physical systems. Morgan Kaufmann. [find](https://scholar.google.com/scholar?q=Daniel+S+Weld+and+Johan+De+Kleer%2E+2013%2E+Readings+in+qualitative+reasoning+about+physical+systems%2E+Morgan+Kaufmann%2E)
15. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In EMNLP.

*Updated 2 November 2026*