---
title: "MSciNLI — paper by Sadat and Caragea (2024)"
description: "The authors build a multi-domain resource from four ACM Digital Library… 2.02 points on MSciNLI. 40 references, cited by 5 papers here."
authors: ["Mobashir Sadat", "Cornelia Caragea"]
year: 2024
venue: "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)"
source: "https://aclanthology.org/2024.naacl-long.90/"
doi: "10.18653/v1/2024.naacl-long.90"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sadat-2024-mscinli-diverse-benchmark-scientific-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sadat-2024-mscinli-diverse-benchmark-scientific-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/) / MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference

# MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference

**Authors:** [Mobashir Sadat](https://md.research.elamur.ai/free/en/authors/mobashir-sadat/) and [Cornelia Caragea](https://md.research.elamur.ai/free/en/authors/cornelia-caragea/) · **Year:** 2024 · **Venue:** [Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2024](https://md.research.elamur.ai/free/en/venues/naacl/2024/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.naacl-long.90/) · [PDF](https://aclanthology.org/2024.naacl-long.90.pdf) · [DOI 10.18653/v1/2024.naacl-long.90](https://doi.org/10.18653/v1/2024.naacl-long.90) · [OpenAlex](https://openalex.org/W4401042607)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [5](https://md.research.elamur.ai/free/en/papers/sadat-2024-mscinli-diverse-benchmark-scientific-natural/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
MSciNLI tackles a limitation of scientific Natural Language Inference: the only earlier dataset, SciNLI, comes entirely from ACL computational-linguistics papers. Because it covers one domain, it cannot serve as a general benchmark or support domain-shift studies. The authors build a multi-domain resource from four ACM Digital Library categories and from NeurIPS papers. They reuse the SciNLI recipe: adjacent sentence pairs whose second sentence opens with a linking phrase receive a distant label from one of four classes (ENTAILMENT, REASONING, CONTRASTING, NEUTRAL). The cue phrase is then deleted. The large training split stays noisy, while the dev and test splits are labelled by hand. Baselines include a BiLSTM, four fine-tuned PLMs, and two prompted LLMs. The dataset turns out harder than SciNLI, and LLMs lag well behind fine-tuned models. Testing on an unseen domain lowers scores. Supervised intermediate training on the combined scientific NLI data helps several downstream scientific classification tasks. The work matters because it provides a diverse, demanding testbed for reasoning over specialised academic text.

## Key points
- MSciNLI has 132,320 sentence pairs from five domains: Hardware, Networks, Software & its Engineering, Security & Privacy, and NeurIPS. Its 4,000 test and 1,000 dev examples are manually annotated.
- Baselines span a BiLSTM, BERT, SciBERT, RoBERTa and XLNet (fine-tuned), plus Llama-2 and Mistral (zero-shot and few-shot prompting).
- RoBERTa scores best among PLMs at 77.21% Macro F1, while the top LLM result (Llama-2) is only 51.77%. The authors note that prompt design strongly affects LLM scores.
- Cross-dataset tests are asymmetric:
  - A SciNLI-trained model loses 2.02 points on MSciNLI.
  - An MSciNLI-trained model loses only 1.34 points on SciNLI, whose in-dataset score is 78.08%.
  - Training on the union, MSciNLI+, gives the best results on both.
- As an intermediate task before fine-tuning RoBERTa, MSciNLI+ improves all three downstream tasks:
  - SciHTC rises from 52.59 to 53.47.
  - PaperField rises from 73.66 to 74.09.
  - ACL-ARC rises from 69.57 to 73.04.
  - MNLI, SciNLI alone, and MLM-only training on MSciNLI+ give no gain.

## Interesting ideas
- Discourse connectives such as "therefore" or "in contrast" act as free relation labels, and removing them from the hypothesis forces the model to infer the relation from content rather than from the cue word.
- The design combines a big, cheaply labelled, noisy training split with a small hand-checked dev/test split, so the noise affects only learning and never the measurement.
- Training on one domain and testing on others in both directions shows that more varied training data generalises better than a single-source set.
- The intermediate-training study includes a control: an MLM objective run on the same texts without labels. This separates the effect of merely seeing the domain text from the effect of the supervised relation signal.
- The label set goes beyond entail/contradict/neutral by adding REASONING and CONTRASTING, which better reflect how sentences in scientific prose actually connect.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (frames): Raw English text could be mined for connective-linked sentence pairs, giving silver data for cause/contrast links between frames, with the connective stripped before checking whether frame-level reasoning recovers the relation.
- **I2** (learn): Silver data agreed by witness parsers can be kept for training, while a small gold set is reserved purely for evaluation so silver errors cannot inflate reported scores.
- **I3** (tree): The parsers could be evaluated on held-out genres in a train-on-A/test-on-B grid, which would show whether treebank mixing actually buys robustness.
- **I4** (learn): Every new training signal could get a matched control that uses the same text without the labels, as a negative control showing that gains come from the supervision itself.
- **I5** (world): Inter-sentence links in the world of a text could carry explicit because/despite relations, letting readers answer why-questions and spot contrasts with an explainable trace.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 61 works · 20,107 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 2,869 works

🧭 Topics (2): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works · [📏 Evaluation & benchmarks](https://md.research.elamur.ai/free/en/topics/evaluation-benchmarks/) · 1,220 works

## Cite

Cite as: Mobashir Sadat and Cornelia Caragea. 2024. MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sadat2024mscinli,
  title = {{MSciNLI: A Diverse Benchmark for Scientific Natural Language Inference}},
  author = {Mobashir Sadat and Cornelia Caragea},
  booktitle = {Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.90},
  url = {https://doi.org/10.18653/v1/2024.naacl-long.90}
}
```

- [← previous MetaMetrics: Calibrating Metrics For Generation Tasks Using Human Preferences](https://md.research.elamur.ai/free/en/papers/winata-2024-metametrics-calibrating-metrics-generation-tasks/)
- [next → Perceptions of Language Technology Failures from South Asian English Speakers](https://md.research.elamur.ai/free/en/papers/holt-2024-perceptions-language-technology-failures-south/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/10/)

## References

40 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620, Hong Kong, China. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Iz+Beltagy%2C+Kyle+Lo%2C+and+Arman+Cohan%2E+2019%2E+SciB%2DERT%3A+A+pretrained+language+model+for+scientific+text%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the)
2. Steven Bird, Robert Dale, Bonnie J. Dorr, Bryan Gibson, Mark T. Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R. Radev, and Yee Fan Tan. 2008. [The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics](https://md.research.elamur.ai/free/en/papers/bird-2008-acl-anthology-reference-corpus-reference/). In Proc. of the 6th International Conference on Language Resources and Evaluation Conference (LREC’08), pages 1755–1759.
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075)
4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
5. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). arXiv preprint arXiv:1705.02364. [↗](https://arxiv.org/abs/1705.02364)
6. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1809.05053)
7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classificati)
8. Pritam Deka, Anna Jurek-Loughrey, et al. 2022. Evidence extraction to validate medical claims in fake news detection. In International Conference on Health Information Science, pages 3–15. Springer. [find](https://scholar.google.com/scholar?q=Pritam+Deka%2C+Anna+Jurek%2DLoughrey%2C+et+al%2E+2022%2E+Evidence+extraction+to+validate+medical+claims+in+fake+news+detection%2E+In+International+Conference+on+Health+Information+Science%2C+pages+3%E2%80%9315%2E+Springer%2E)
9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2017)
11. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. [Measuring massive multitask language understanding](https://md.research.elamur.ai/free/en/papers/hendrycks-2020-measuring-massive-multitask-language-understanding/). In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.2009.03300)
12. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. [↗](https://arxiv.org/abs/2310.06825)
13. David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc-Farland, and Dan Jurafsky. 2018. [Measuring the evolution of a scientific field through citation frames](https://md.research.elamur.ai/free/en/papers/jurgens-2018-measuring-evolution-scientific-field-through/). Transactions of the Association for Computational Linguistics, 6:391–406. [↗](https://doi.org/10.1162/tacl_a_00028)
14. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAAI, volume 17, pages 41– 42. [find](https://scholar.google.com/scholar?q=Tushar+Khot%2C+Ashish+Sabharwal%2C+and+Peter+Clark%2E+2018%2E+Scitail%3A+A+textual+entailment+dataset+from+science+question+answering%2E+In+AAAI%2C+volume+17%2C+pages+41%E2%80%93+42%2E)
15. Diederik P Kingma and Jimmy Ba. 2014. [Adam: A method for stochastic optimization](https://md.research.elamur.ai/free/en/papers/kingma-2014-adam-method-stochastic-optimization/). arXiv preprint arXiv:1412.6980. [↗](https://arxiv.org/abs/1412.6980)
16. Dan Klein and Christopher D. Manning. 2003. [Accurate unlexicalized parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423–430, Sapporo, Japan. Association for Computational Linguistics. [↗](https://doi.org/10.3115/1075096.1075150)
17. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), pages 54–61, Suntec City, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=J+Richard+Landis+and+Gary+G+Koch%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+biometrics%2C+pages+159%E2%80%93174%2E+and+Citation+Analysis+for+Scholarly+Digital+Libraries+%28NLPIR4DL%29%2C+pages+5)
18. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. [Multi-task deep neural networks for natural language understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4487–4496, Florence, Italy. Association for Computational Linguistics.
19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. [SQuAD: 100,000+ questions for machine comprehension of text](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/). In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d16-1264)
20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
21. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. [A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA). [↗](https://doi.org/10.63317/39qdhuevzbqa)
22. Alejandro Martín, Javier Huertas-Tato, Álvaro Huertas-García, Guillermo Villar-Rodríguez, and David Camacho. 2022. Facter-check: Semi-automated fact-checking through semantic similarity and natural language inference. Knowledge-Based Systems, 251:109265. [↗](https://doi.org/10.1016/j.knosys.2022.109265)
23. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019. [Adversarial nli: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). arXiv preprint arXiv:1910.14599. [↗](https://arxiv.org/abs/1910.14599)
24. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. [Adversarial NLI: A new benchmark for natural language understanding](https://md.research.elamur.ai/free/en/papers/nie-2020-adversarial-nli-new-benchmark-natural/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.acl-main.441)
25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. [Glove: Global vectors for word representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. [↗](https://doi.org/10.3115/v1/d14-1162)
26. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. [Intermediate-task transfer learning with pretrained language models: When and why does it work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231–5247, Online. Association for Computational Linguistics.
27. Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian. 2009. The ACL Anthology network. In Proceedings of the 2009 Workshop on Text Mobashir Sadat and Cornelia Caragea. 2022a. Hierarchical multi-label classification of scientific documents. In Proceedings of The 2022 Conference on Empirical Methods in Natural Language Processing (Volume 1: Long Papers), Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dragomir+R%2E+Radev%2C+Pradeep+Muthukrishnan%2C+and+Vahed+Qazvinian%2E+2009%2E+The+ACL+Anthology+network%2E+In+Proceedings+of+the+2009+Workshop+on+Text+Mobashir+Sadat+and+Cornelia+Caragea%2E+2022a%2E+Hierarchical+mul)
28. Mobashir Sadat and Cornelia Caragea. 2022b. [SciNLI: A corpus for natural language inference on scientific text](https://md.research.elamur.ai/free/en/papers/sadat-2022-scinli-corpus-natural-language-inference/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7399–7409, Dublin, Ireland. Association for Computational Linguistics.
29. Fariba Sadeghi, Amir Jalaly Bidgoly, and Hossein Amirkhani. 2022. Fake news detection on social media using a natural language inference approach. Multimedia Tools and Applications, pages 1–21. [↗](https://doi.org/10.1007/s11042-022-12428-8)
30. Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9275–9293, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Swabha+Swayamdipta%2C+Roy+Schwartz%2C+Nicholas+Lourie%2C+Yizhong+Wang%2C+Hannaneh+Hajishirzi%2C+Noah+A%2E+Smith%2C+and+Yejin+Choi%2E+2020%2E+Dataset+cartography%3A+Mapping+and+diagnosing+datasets+with+training+dynamics%2E+)
31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/). arXiv preprint arXiv:2307.09288. [↗](https://arxiv.org/abs/2307.09288)
32. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
33. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/w18-5446)
34. Chenglin Wang, Yucheng Zhou, Guodong Long, Xiaodong Wang, and Xiaowei Xu. 2022. Unsupervised knowledge graph construction and event-centric knowledge infusion for scientific nli. arXiv preprint arXiv:2210.15248. [↗](https://arxiv.org/abs/2210.15248)
35. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018a. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
36. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018b. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-1101)
37. Jinxuan Wu, Wenhan Chao, Xian Zhou, and Zhunchen Luo. 2023. Characterizing and verifying scientific claims: Qualitative causal structure is all you need. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13428–13439, Singapore. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jinxuan+Wu%2C+Wenhan+Chao%2C+Xian+Zhou%2C+and+Zhunchen+Luo%2E+2023%2E+Characterizing+and+verifying+scientific+claims%3A+Qualitative+causal+structure+is+all+you+need%2E+In+Proceedings+of+the+2023+Conference+on+Empir)
38. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in neural information processing systems, pages 5753–5763. [↗](https://doi.org/10.48550/arxiv.1906.08237)
39. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint arXiv:2302.10198. [↗](https://arxiv.org/abs/2302.10198)
40. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, pages 19–27. Class Linking Phrases C ONTRASTING ‘However’, ‘On the other hand’, ‘In contrast’, ‘On the contrary’ R EASONING ‘Therefore’, ‘Thus’, ‘Consequently’, ‘As a result’, ‘As a consequence’, ‘From here, we can infer’ E NTAILMENT ‘Specifically’, ‘Precisely’, ‘In particular’, ‘Particularly’, ‘That is’, ‘In other words’ *Table 11: Linking phrases used to extract sentence pairs and their corresponding classes.*

*Updated 1 November 2026*