--- title: "Reasoning Over Paragraph Effects in Situations — Lin 2019" description: "ROPES is a reading comprehension benchmark from the Allen Institute for AI." authors: ["Kevin Lin", "Oyvind Tafjord", "Peter Clark", "Matt Gardner"] year: 2019 venue: "Proceedings of the 2nd Workshop on Machine Reading for Question Answering" source: "https://aclanthology.org/D19-5808/" doi: "10.18653/v1/d19-5808" arxiv: "1908.05852" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/lin-2019-reasoning-over-paragraph-effects-situations/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/lin-2019-reasoning-over-paragraph-effects-situations/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / Reasoning Over Paragraph Effects in Situations # Reasoning Over Paragraph Effects in Situations **Authors:** [Kevin Lin](https://md.research.elamur.ai/free/en/authors/kevin-lin/), [Oyvind Tafjord](https://md.research.elamur.ai/free/en/authors/oyvind-tafjord/), [Peter Clark](https://md.research.elamur.ai/free/en/authors/peter-clark/), [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/) · **Year:** 2019 · **Venue:** Proceedings of the 2nd Workshop on Machine Reading for Question Answering · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/D19-5808/) · [PDF](https://aclanthology.org/D19-5808.pdf) · [DOI 10.18653/v1/d19-5808](https://doi.org/10.18653/v1/d19-5808) · [arXiv 1908.05852](https://arxiv.org/abs/1908.05852) · [OpenAlex](https://openalex.org/W2987553933) **Citations:** 92 citations (openalex-snapshot · 2026-10-14) · [79](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary ROPES is a reading comprehension benchmark from the Allen Institute for AI. It tests whether a system can take a cause–effect relationship described in an expository text and apply it to an unfamiliar scenario. Most existing QA datasets reward locating local facts in a passage, which current models already handle at near-human level. To build ROPES, the authors scraped paragraphs from science textbooks (CK12, OpenStax) and Wikipedia that contained causal or qualitative cue words, then filtered them by hand. Amazon Mechanical Turk workers wrote a new situation for each chosen passage, plus questions whose span answers require linking the situation to the background. Workers were encouraged to write minimal question pairs to discourage shortcuts. The train, dev and test splits were separated by annotator to limit worker bias. The resulting 14,322 questions were used to evaluate RoBERTa baselines. The strongest system, RoBERTa LARGE additionally fine-tuned on RACE, reached 61.6% F1 on test, compared with 89.0% for expert humans. Since most questions offer two plausible options, this is barely above chance. The result exposes a clear gap in applying knowledge from reading to new contexts. ## Key points - Dataset size: 14,322 questions built on 588 worker-selected background passages, drawn from a pool of over 1,000. Topics are mostly physical science (49%) and life science (45%), with some economics (5%). - Splits are separated by annotator: 7 workers for train (10,924 questions), 2 for dev (1,688) and 2 for test (1,710). Splitting by situation gave inflated scores because of annotator bias. - In a 100-question sample, 71% of questions were effect comparisons and 15% were cause comparisons. Grounding the relation was explicit in 67% of cases, needed common sense in 13%, and required bridging a lexical gap in 20%. - Test results: RoBERTa BASE scored 45.5 F1, RoBERTa LARGE 61.1 F1, and RoBERTa LARGE + RACE 61.6 F1. Humans scored 82.7 EM and 89.0 F1, measured on 400 questions. - Removing the background passage cost RoBERTa LARGE little on test (60.4 F1). This suggests models rely on artifacts rather than truly using the background text. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 79 papers of our library cite it (of the 140884 whose bibliographies we read); 9 of them have a page here, 70 not yet. - [Transformers as Soft Reasoners over Language](https://md.research.elamur.ai/free/en/papers/clark-2020-transformers-soft-reasoners-over-language/) - [Evaluating Models' Local Decision Boundaries via Contrast Sets](https://md.research.elamur.ai/free/en/papers/gardner-2020-evaluating-models-local-decision-boundaries/) - [Evaluating Open-Domain Question Answering in the Era of Large Language Models](https://md.research.elamur.ai/free/en/papers/kamalloo-2023-evaluating-open-domain-question-answering/) - [UNIFIEDQA: Crossing Format Boundaries with a Single QA System](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/) - [TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions](https://md.research.elamur.ai/free/en/papers/ning-2020-torque-reading-comprehension-dataset-temporal/) - [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://md.research.elamur.ai/free/en/papers/sanh-2021-multitask-prompted-training-enables-zero/) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/) - [Galactica: A Large Language Model for Science](https://md.research.elamur.ai/free/en/papers/taylor-2022-galactica-large-language-model-science/) - [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](https://md.research.elamur.ai/free/en/papers/ustun-2024-aya-model-instruction-finetuned-open/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{lin2019reasoning, title = {{Reasoning Over Paragraph Effects in Situations}}, author = {Lin, Kevin and Tafjord, Oyvind and Clark, Peter and Gardner, Matt}, booktitle = {Proceedings of the 2nd Workshop on Machine Reading for Question Answering}, year = {2019}, doi = {10.18653/v1/d19-5808}, eprint = {1908.05852}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/d19-5808} } ``` - [← previous Real-Time Open-Domain Question Answering with Dense-Sparse Phrase Index](https://md.research.elamur.ai/free/en/papers/seo-2019-real-time-open-domain-question/) - [next → Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches](https://md.research.elamur.ai/free/en/papers/storks-2019-recent-advances-natural-language-inference/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) ## References 11 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP. [find](https://scholar.google.com/scholar?q=Matthew+Richardson%2C+Christopher+J%2E+C%2E+Burges%2C+and+Erin+Renshaw%2E+2013%2E+Mctest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/) 2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2E) 3. Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. arXiv preprint arXiv:1809.01494. [find](https://scholar.google.com/scholar?q=Marzieh+Saeidi%2C+Max+Bartolo%2C+Patrick+Lewis%2C+Sameer+Singh%2C+Tim+Rockta%CC%88schel%2C+Mike+Sheldon%2C+Guillaume+Bouchard%2C+and+Sebastian+Riedel%2E+2018%2E+Interpretation+of+natural+language+rules+in+conversational+mac) [our card](https://md.research.elamur.ai/free/en/papers/saeidi-2018-interpretation-natural-language-rules-conversational/) 4. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858. [find](https://scholar.google.com/scholar?q=Danqi+Chen%2C+Jason+Bolton%2C+and+Christopher+D+Manning%2E+2016%2E+A+thorough+examination+of+the+cnn%2Fdaily+mail+reading+comprehension+task%2E+arXiv+preprint+arXiv%3A1606%2E02858%2E) [our card](https://md.research.elamur.ai/free/en/papers/chen-2016-thorough-examination-cnn-daily-mail/) 5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. Lecture Notes in Computer Science, pages 177–190. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+pascal+recognising+textual+entailment+challenge%2E+Lecture+Notes+in+Computer+Science%2C+pages+177%E2%80%93190%2E) 6. Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. 2019. Quarel: A dataset and models for answering questions about qualitative relationships. In AAAI. [find](https://scholar.google.com/scholar?q=Oyvind+Tafjord%2C+Peter+Clark%2C+Matt+Gardner%2C+Wen%2Dtau+Yih%2C+and+Ashish+Sabharwal%2E+2019%2E+Quarel%3A+A+dataset+and+models+for+answering+questions+about+qualitative+relationships%2E+In+AAAI%2E) 7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2E) 8. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 189–198. [find](https://scholar.google.com/scholar?q=Wenhui+Wang%2C+Nan+Yang%2C+Furu+Wei%2C+Baobao+Chang%2C+and+Ming+Zhou%2E+2017%2E+Gated+self%2Dmatching+networks+for+reading+comprehension+and+question+answering%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Asso) [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/) 9. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In NAACL. [find](https://scholar.google.com/scholar?q=Dheeru+Dua%2C+Yizhong+Wang%2C+Pradeep+Dasigi%2C+Gabriel+Stanovsky%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2019%2E+DROP%3A+A+reading+comprehension+benchmark+requiring+discrete+reasoning+over+paragraphs%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/dua-2019-drop-reading-comprehension-benchmark-requiring/) 10. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan R. Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In EMNLP. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Peng+Qi%2C+Saizheng+Zhang%2C+Yoshua+Bengio%2C+William+W%2E+Cohen%2C+Ruslan+R%2E+Salakhutdinov%2C+and+Christopher+D%2E+Manning%2E+2018%2E+Hotpotqa%3A+A+dataset+for+diverse%2C+explainable+multi%2Dhop+question+answer) [our card](https://md.research.elamur.ai/free/en/papers/yang-2018-hotpotqa-dataset-diverse-explainable-multi/) 11. Mor Geva, Yoav Goldberg, and Jonathan Berant. 2019. Are we modeling the task or the annotator? an investigation of annotator bias in natural language understanding datasets. EMNLP. [find](https://scholar.google.com/scholar?q=Mor+Geva%2C+Yoav+Goldberg%2C+and+Jonathan+Berant%2E+2019%2E+Are+we+modeling+the+task+or+the+annotator%3F+an+investigation+of+annotator+bias+in+natural+language+understanding+datasets%2E+EMNLP%2E)