---
title: "Logic-Guided Data Augmentation and Regularization for"
description: "Comparison questions ask how two entities or events relate, for example which is larger or what… 31 references, cited by 48 papers here."
authors: ["Akari Asai", "Hannaneh Hajishirzi"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.499/"
doi: "10.18653/v1/2020.acl-main.499"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/asai-2020-logic-guided-data-augmentation-regularization/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/asai-2020-logic-guided-data-augmentation-regularization/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Logic-Guided Data Augmentation and Regularization for Consistent Question Answering

# Logic-Guided Data Augmentation and Regularization for Consistent Question Answering

**Authors:** [Akari Asai](https://md.research.elamur.ai/free/en/authors/akari-asai/) and [Hannaneh Hajishirzi](https://md.research.elamur.ai/free/en/authors/hannaneh-hajishirzi/) · **Year:** 2020 · **Venue:** [Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2020](https://md.research.elamur.ai/free/en/venues/acl/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.499/) · [PDF](https://aclanthology.org/2020.acl-main.499.pdf) · [DOI 10.18653/v1/2020.acl-main.499](https://doi.org/10.18653/v1/2020.acl-main.499) · [OpenAlex](https://openalex.org/W3035438620)

**Citations:** 63 citations (OpenAlex, 2026-10-14) · [48](https://md.research.elamur.ai/free/en/papers/asai-2020-logic-guided-data-augmentation-regularization/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Comparison questions ask how two entities or events relate, for example which is larger or what effect a cause has. Neural QA models such as RoBERTa often answer them inconsistently. A model may give the same answer to a question and its antonym-flipped version, or break a chain of cause-effect reasoning. Such datasets are also costly to annotate, so they are small.

Asai and Hajishirzi address this with two logic-driven components. The first is data augmentation that creates new labeled examples using symmetry and transitivity rules. Symmetric examples come from antonym substitution or from adding or removing negation. Transitive examples come from chaining linked cause-effect questions. The second is a consistency regularizer added to the task loss, which penalizes predictions that violate these rules.

The method does not depend on any particular model. It was tested on three formats: WIQA (classification), QuaRel (multiple choice) and HotpotQA (span extraction). It raised accuracy, set new state-of-the-art results on WIQA and QuaRel, and sharply reduced inconsistent predictions. It also worked well with little labeled data.

## Key points
- Applied to RoBERTa, the method improves results by 1.0 points on WIQA, 5.0 on QuaRel and 2.5 on HotpotQA. It surpasses the previous state of the art on WIQA by 4.7 points and on QuaRel by 8.4.
- Consistency violations fall by 58% on HotpotQA. On WIQA, symmetric violations drop from 13.9% to 8.3% and transitive violations from 10.0% to 2.5%.
- With only 20% of the training data, gains over RoBERTa exceed 12 points on WIQA and 14 points on QuaRel. This matches a strong baseline trained on the full data.
- In the ablation, a conventional WordNet-based paraphrasing augmentation without logic rules gave no notable gains. Combining logic-guided augmentation with regularization worked best.
- Augmented samples (24,715 for WIQA, 836 for QuaRel and 3,538 for HotpotQA) are drawn once before training. Code and data are available at github.com/AkariAsai/logic_guided_qa.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [❓ Question answering](https://md.research.elamur.ai/free/en/topics/question-answering/) · 243 works

## Cite

Cite as: Akari Asai and Hannaneh Hajishirzi. 2020. Logic-Guided Data Augmentation and Regularization for Consistent Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{asai2020logic,
  title = {{Logic-Guided Data Augmentation and Regularization for Consistent Question Answering}},
  author = {Akari Asai and Hannaneh Hajishirzi},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.499},
  url = {https://doi.org/10.18653/v1/2020.acl-main.499}
}
```

- [← previous Learning from Task Descriptions](https://md.research.elamur.ai/free/en/papers/weller-2020-learning-task-descriptions/)
- [next → XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization](https://md.research.elamur.ai/free/en/papers/raganato-2020-xl-wic-multilingual-benchmark-evaluating/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/)

## References

31 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. 2019. AutoAugment: Learning augmentation strategies from data. In CVPR. [find](https://scholar.google.com/scholar?q=Ekin+D%2E+Cubuk%2C+Barret+Zoph%2C+Dandelion+Mane%2C+Vijay+Vasudevan%2C+and+Quoc+V%2E+Le%2E+2019%2E+AutoAugment%3A+Learning+augmentation+strategies+from+data%2E+In+CVPR%2E)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In NAACL.
3. Ming Ding, Chang Zhou, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. 2019. Cognitive graph for multi-hop reading comprehension at scale. In ACL. [find](https://scholar.google.com/scholar?q=Ming+Ding%2C+Chang+Zhou%2C+Chang+Zhou%2C+Qibin+Chen%2C+Hongxia+Yang%2C+and+Jie+Tang%2E+2019%2E+Cognitive+graph+for+multi%2Dhop+reading+comprehension+at+scale%2E+In+ACL%2E)
4. Jay Yoon Lee, Sanket Vaibhav Mehta, Michael Wick, Jean-Baptiste Tristan, and Jaime Carbonell. 2019. Gradient-based inference for networks with output constraints. In AAAI. [find](https://scholar.google.com/scholar?q=Jay+Yoon+Lee%2C+Sanket+Vaibhav+Mehta%2C+Michael+Wick%2C+Jean%2DBaptiste+Tristan%2C+and+Jaime+Carbonell%2E+2019%2E+Gradient%2Dbased+inference+for+networks+with+output+constraints%2E+In+AAAI%2E)
5. Tao Li, Vivek Cupta, Maitrey Mehta, and Vivek Srikumar. 2019. A logic-driven framework for consistency of neural models. In EMNLP. [find](https://scholar.google.com/scholar?q=Tao+Li%2C+Vivek+Cupta%2C+Maitrey+Mehta%2C+and+Vivek+Srikumar%2E+2019%2E+A+logic%2Ddriven+framework+for+consistency+of+neural+models%2E+In+EMNLP%2E)
6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a)
7. George A Miller. 1995. [Wordnet: a lexical database for english](https://md.research.elamur.ai/free/en/papers/miller-1994-wordnet/). Communications of the ACM.
8. Xinya Du, Bhavana Dalvi, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, and Claire Cardie. 2019. Be consistent! improving procedural text comprehension using label consistency. In NAACL. [find](https://scholar.google.com/scholar?q=Xinya+Du%2C+Bhavana+Dalvi%2C+Niket+Tandon%2C+Antoine+Bosselut%2C+Wen%2Dtau+Yih%2C+Peter+Clark%2C+and+Claire+Cardie%2E+2019%2E+Be+consistent%21+improving+procedural+text+comprehension+using+label+consistency%2E+In+NAACL%2E)
9. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019a. [Compositional questions do not necessitate multi-hop reasoning](https://md.research.elamur.ai/free/en/papers/min-2019-compositional-questions-do-not-necessitate/). In ACL.
10. Ali Elkahky, Kellie Webster, Daniel Andor, and Emily Pitler. 2018. A challenge set and methods for noun-verb ambiguity. In EMNLP. [find](https://scholar.google.com/scholar?q=Ali+Elkahky%2C+Kellie+Webster%2C+Daniel+Andor%2C+and+Emily+Pitler%2E+2018%2E+A+challenge+set+and+methods+for+noun%2Dverb+ambiguity%2E+In+EMNLP%2E)
11. Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2019b. [Multi-hop reading comprehension through question decomposition and rescoring](https://md.research.elamur.ai/free/en/papers/min-2019-multi-hop-reading-comprehension-through/). In ACL.
12. Kuzman Ganchev, Jennifer Gillenwater, Ben Taskar, et al. 2010. Posterior regularization for structured latent variable models. Journal of Machine Learning Research, 11(Jul):2001–2049. [find](https://scholar.google.com/scholar?q=Kuzman+Ganchev%2C+Jennifer+Gillenwater%2C+Ben+Taskar%2C+et+al%2E+2010%2E+Posterior+regularization+for+structured+latent+variable+models%2E+Journal+of+Machine+Learning+Research%2C+11%28Jul%29%3A2001%E2%80%932049%2E)
13. Pasquale Minervini and Sebastian Riedel. 2018. Adversarially regularising neural NLI models to integrate logical background knowledge. In ACL. [find](https://scholar.google.com/scholar?q=Pasquale+Minervini+and+Sebastian+Riedel%2E+2018%2E+Adversarially+regularising+neural+NLI+models+to+integrate+logical+background+knowledge%2E+In+ACL%2E)
14. Regina Paxton Gazes, Nicholas W Chee, and Robert R Hampton. 2012. Cognitive mechanisms for transitive inference performance in rhesus monkeys: Measuring the influence of associative strength and inferred order. Journal of Experimental Psychology: Animal Behavior Processes, 38(4):331. [find](https://scholar.google.com/scholar?q=Regina+Paxton+Gazes%2C+Nicholas+W+Chee%2C+and+Robert+R+Hampton%2E+2012%2E+Cognitive+mechanisms+for+transitive+inference+performance+in+rhesus+monkeys%3A+Measuring+the+influence+of+associative+strength+and+infer)
15. Tejas Gokhale, Pratyay Banerjee, Chitta Baral, and Yezhou Yang. 2020. VQA-LOL: Visual question answering under the lens of logic. arXiv:2002.08325. — [arXiv:2002.08325](https://arxiv.org/abs/2002.08325) [find](https://scholar.google.com/scholar?q=Tejas+Gokhale%2C+Pratyay+Banerjee%2C+Chitta+Baral%2C+and+Yezhou+Yang%2E+2020%2E+VQA%2DLOL%3A+Visual+question+answering+under+the+lens+of+logic%2E+arXiv%3A2002%2E08325%2E+%E2%80%94+%5BarXiv%3A2002%2E08325%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2002%2E0832)
16. He He, Sheng Zha, and Haohan Wang. 2019. Unlearn dataset bias for natural language inference by fitting the residual. In EMNLP Workshop on DeepLo. [find](https://scholar.google.com/scholar?q=He+He%2C+Sheng+Zha%2C+and+Haohan+Wang%2E+2019%2E+Unlearn+dataset+bias+for+natural+language+inference+by+fitting+the+residual%2E+In+EMNLP+Workshop+on+DeepLo%2E)
17. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. [TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/). In ACL.
18. Dongyeop Kang, Tushar Khot, Ashish Sabharwal, and Eduard Hovy. 2018. AdvEntuRe: Adversarial training for textual entailment with knowledge-guided examples. In ACL. [find](https://scholar.google.com/scholar?q=Dongyeop+Kang%2C+Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Eduard+Hovy%2E+2018%2E+AdvEntuRe%3A+Adversarial+training+for+textual+entailment+with+knowledge%2Dguided+examples%2E+In+ACL%2E)
19. Scott Kirkpatrick, C Daniel Gelatt, and Mario P Vecchi. 1983. Optimization by simulated annealing. science. [find](https://scholar.google.com/scholar?q=Scott+Kirkpatrick%2C+C+Daniel+Gelatt%2C+and+Mario+P+Vecchi%2E+1983%2E+Optimization+by+simulated+annealing%2E+science%2E)
20. Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. Technical report, Citeseer. [find](https://scholar.google.com/scholar?q=Alex+Krizhevsky%2C+Geoffrey+Hinton%2C+et+al%2E+2009%2E+Learning+multiple+layers+of+features+from+tiny+images%2E+Technical+report%2C+Citeseer%2E)
21. Arindam Mitra, Chitta Baral, Aurgho Bhattacharjee, and Ishan Shrivastava. 2019. A generate-validate approach to answering questions about qualitative relationships. arXiv:1908.03645. — [arXiv:1908.03645](https://arxiv.org/abs/1908.03645) [find](https://scholar.google.com/scholar?q=Arindam+Mitra%2C+Chitta+Baral%2C+Aurgho+Bhattacharjee%2C+and+Ishan+Shrivastava%2E+2019%2E+A+generate%2Dvalidate+approach+to+answering+questions+about+qualitative+relationships%2E+arXiv%3A1908%2E03645%2E+%E2%80%94+%5BarXiv%3A1908%2E036)
22. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). arXiv:1904.01038.
23. Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. 2019. [SpecAugment: A simple data augmentation method for automatic speech recognition](https://md.research.elamur.ai/free/en/papers/park-2019-specaugment-simple-data-augmentation-method/). arXiv:1904.08779.
24. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In EMNLP.
25. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. [What makes reading comprehension questions easier?](https://md.research.elamur.ai/free/en/papers/sugawara-2018-what-makes-reading-comprehension-questions/) In EMNLP.
26. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. QuaRel: A dataset and models for answering questions about qualitative relationships. In AAAI. [find](https://scholar.google.com/scholar?q=Oyvind+Tafjord%2C+Peter+Clark%2C+Matt+Gardner%2C+Wen%2Dtau+Yih%2C+and+Ashish+Sabharwal%2E+2019%2E+QuaRel%3A+A+dataset+and+models+for+answering+questions+about+qualitative+relationships%2E+In+AAAI%2E)
27. Niket Tandon, Bhavana Dalvi Mishra, Keisuke Sakaguchi, Antoine Bosselut, and Peter Clark. 2019. WIQA: A dataset for ”what if...” reasoning over procedural text. In EMNLP. [find](https://scholar.google.com/scholar?q=Niket+Tandon%2C+Bhavana+Dalvi+Mishra%2C+Keisuke+Sakaguchi%2C+Antoine+Bosselut%2C+and+Peter+Clark%2E+2019%2E+WIQA%3A+A+dataset+for+%E2%80%9Dwhat+if%2E%2E%2E%E2%80%9D+reasoning+over+procedural+text%2E+In+EMNLP%2E)
28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). arXiv:1910.03771.
29. Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. 2019. Adversarial examples improve image recognition. arXiv:1911.09665. — [arXiv:1911.09665](https://arxiv.org/abs/1911.09665) [find](https://scholar.google.com/scholar?q=Cihang+Xie%2C+Mingxing+Tan%2C+Boqing+Gong%2C+Jiang+Wang%2C+Alan+Yuille%2C+and+Quoc+V+Le%2E+2019%2E+Adversarial+examples+improve+image+recognition%2E+arXiv%3A1911%2E09665%2E+%E2%80%94+%5BarXiv%3A1911%2E09665%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1911%2E0)
30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. [HotpotQA: A dataset for diverse, explainable multi-hop question answering](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/). In EMNLP.
31. Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. [QANet: Combining local convolution with global self-attention for reading comprehension](https://md.research.elamur.ai/free/en/papers/yu-2018-qanet-combining-local-convolution-global/). In ICLR.

*Updated 24 October 2026*