---
title: "SciNLI: A Corpus for Natural Language Inference on Scientific Text"
description: "The authors build SciNLI, an inference corpus drawn from NLP and computational… 78.23% accuracy. 41 references, cited by 23 papers here."
authors: ["Mobashir Sadat", "Cornelia Caragea"]
year: 2022
venue: "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2022.acl-long.511/"
doi: "10.18653/v1/2022.acl-long.511"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/sadat-2022-scinli-corpus-natural-language-inference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/sadat-2022-scinli-corpus-natural-language-inference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / SciNLI: A Corpus for Natural Language Inference on Scientific Text

# SciNLI: A Corpus for Natural Language Inference on Scientific Text

**Authors:** Mobashir Sadat and [Cornelia Caragea](https://md.research.elamur.ai/free/en/authors/cornelia-caragea/) · **Year:** 2022 · **Venue:** [Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2022](https://md.research.elamur.ai/free/en/venues/acl/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.acl-long.511/) · [PDF](https://aclanthology.org/2022.acl-long.511.pdf) · [DOI 10.18653/v1/2022.acl-long.511](https://doi.org/10.18653/v1/2022.acl-long.511) · [OpenAlex](https://openalex.org/W4226065433)

**Citations:** 20 citations (OpenAlex, 2026-10-14) · 23 in our library, of the 141,840 papers whose bibliographies we read

## Summary
Most NLI benchmarks are built from everyday language such as image captions, conversations and travel guides. They therefore say little about how well models reason over the dense, formal prose of research articles. The authors build SciNLI, an inference corpus drawn from NLP and computational linguistics papers. They observe that writers often open a sentence with a connective that signals how it relates to the previous one. These connectives serve as distant supervision: each adjacent sentence pair is labeled by its opening phrase, and the phrase is then removed. Besides ENTAILMENT and NEUTRAL, the corpus adds two classes typical of scientific writing. CONTRASTING widens contradiction to cover comparison, criticism and limitation. REASONING marks a first sentence that gives the cause or condition for a conclusion in the second. Training pairs stay noisy, while development and test pairs are checked by hand. Feature-based, neural and pre-trained transformer baselines all leave a clear gap, with XLNet reaching only about 78% Macro F1. SciNLI thus offers a harder, domain-specific benchmark for language understanding.

## Key points
- SciNLI has 107,412 sentence pairs and four labels: CONTRASTING, REASONING, ENTAILMENT, NEUTRAL. Data and code are released at github.com/msadat3/SciNLI.
- Labels come from sentence-initial linking phrases:
  - "However" and "In contrast" signal CONTRASTING.
  - "Therefore", "Thus" and "As a result" signal REASONING.
  - "Specifically", "In particular" and "In other words" signal ENTAILMENT.
- 6,000 pairs are manually annotated, split into 2,000 for development and 4,000 for testing.
- Baselines include lexical/syntactic feature models, BiLSTM, CBOW, CNN, BERT, SciBERT, RoBERTa and XLNet.
  - XLNet is best, at 78.18% Macro F1 and 78.23% accuracy.
  - SciBERT consistently beats BERT (77.66 vs 75.36 F1).
- Feeding only the second sentence sharply lowers scores (BERT 54.56 F1, SciBERT 58.16), so the labels are not guessable from hypothesis artifacts alone.

## Interesting ideas
- Discourse connectives at sentence starts act as free, cheap labels for the relation to the previous sentence; deleting the cue forces a model to infer the relation from content alone.
- The label set is extended beyond the classic entailment/contradiction/neutral triad with CONTRASTING and REASONING, because those relations dominate argumentative text.
- A hypothesis-only baseline serves as a negative control, showing whether labels leak through one side of the pair.
- NEUTRAL pairs are built by sampling in several ways; pairs where one sentence comes from a related-class pair prove harder than fully random pairs. This makes it possible to grade negatives by difficulty.
- Error analysis of 100 test examples traces many failures to missing domain knowledge and to long sentences. Such manual inspection helps decide where to add knowledge rather than capacity.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Connective-marked sentence pairs from English text could become silver data for training and checking inter-sentence relation detection, with the cue hidden from the learner.
- **I2** (world): Event ordering and state tracking could record explicit "because/therefore" and "in contrast" links between propositions, so answers can cite why one fact follows from another.
- **I3** (learn): Any Mova inference or QA evaluation should include a run that sees only part of the input, to show that the full pipeline really uses the premise.
- **I4** (answer): Distractors for school-test or fable QA could be drawn from semantically related sentences rather than random ones, giving tougher and more honest tests.
- **I5** (frames): Frame extraction on long, clause-heavy sentences deserves targeted diagnostics, and missing background facts could be flagged as such instead of producing a wrong frame.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Chicago](https://md.research.elamur.ai/free/en/universities/university-of-illinois-chicago/) — University, Chicago, United States · 38 works · 19,844 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 510 works

## Cite

Cite as: Mobashir Sadat and Cornelia Caragea. 2022. SciNLI: A Corpus for Natural Language Inference on Scientific Text. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{sadat2022scinli,
  title = {{SciNLI: A Corpus for Natural Language Inference on Scientific Text}},
  author = {Mobashir Sadat and Cornelia Caragea},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.511},
  url = {https://doi.org/10.18653/v1/2022.acl-long.511}
}
```

- [← previous Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations](https://md.research.elamur.ai/free/en/papers/long-2022-facilitating-contrastive-learning-discourse-relational/)
- [next → ArMIS - The Arabic Misogyny and Sexism Corpus with Annotator Subjective Disagreements](https://md.research.elamur.ai/free/en/papers/almanea-2022-armis-arabic-misogyny-sexism-corpus/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

41 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Beltagy, I.; Lo, K.; and Cohan, A. 2019. [SciBERT: A Pretrained Language Model for Scientific Text](https://md.research.elamur.ai/free/en/papers/beltagy-2019-scibert-pretrained-language-model-scientific/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3615–3620. Hong Kong, China: Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1903.10676)
2. Bird, S.; Dale, R.; Dorr, B. J.; Gibson, B.; Joseph, M. T.; Kan, M.-Y.; Lee, D.; Powley, B.; Radev, D. R.; and Tan, Y. F. 2008. [The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics](https://md.research.elamur.ai/free/en/papers/bird-2008-acl-anthology-reference-corpus-reference/). In Proc. of the 6th International Conference on Language Resources and Evaluation Conference (LREC’08), 1755–1759.
3. Bowman, S. R.; Angeli, G.; Potts, C.; and Manning, C. D. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, 632–642. Lisbon, Portugal: Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1508.05326)
4. Conneau, A.; Kiela, D.; Schwenk, H.; Barrault, L.; and Bordes, A. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). arXiv preprint arXiv:1705.02364. [↗](https://arxiv.org/abs/1705.02364)
5. Dagan, I.; Glickman, O.; and Magnini, B. 2005. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges Workshop, 177– 190. Springer. Louisiana: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%3B+Glickman%2C+O%2E%3B+and+Magnini%2C+B%2E+2005%2E+The+PASCAL+recognising+textual+entailment+challenge%2E+In+Machine+Learning+Challenges+Workshop%2C+177%E2%80%93+190%2E+Springer%2E+Louisiana%3A+Association+for+Computationa)
6. Hall, D.; Jurafsky, D.; and Manning, C. D. 2008. Studying the history of ideas using topic models. In Proceedings of the conference on empirical methods in natural language processing, 363–371. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Hall%2C+D%2E%3B+Jurafsky%2C+D%2E%3B+and+Manning%2C+C%2E+D%2E+2008%2E+Studying+the+history+of+ideas+using+topic+models%2E+In+Proceedings+of+the+conference+on+empirical+methods+in+natural+language+processing%2C+363%E2%80%93371%2E+Associ)
7. Hobbs, J. R. 1978. Why is discourse coherent. Technical report, SRI INTERNATIONAL MENLO PARK CA. [find](https://scholar.google.com/scholar?q=Hobbs%2C+J%2E+R%2E+1978%2E+Why+is+discourse+coherent%2E+Technical+report%2C+SRI+INTERNATIONAL+MENLO+PARK+CA%2E)
8. Hochreiter, S.; and Schmidhuber, J. 1997. Long short-term memory. Neural computation, 9(8): 1735– 1780. [↗](https://doi.org/10.1162/neco.1997.9.8.1735)
9. Jernite, Y.; Bowman, S. R.; and Sontag, D. 2017. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557. [↗](https://arxiv.org/abs/1705.00557)
10. Khot, T.; Sabharwal, A.; and Clark, P. 2018. SciTaiL: A Textual Entailment Dataset from Science Question Answering. In AAAI, volume 17, 41–42. [find](https://scholar.google.com/scholar?q=Khot%2C+T%2E%3B+Sabharwal%2C+A%2E%3B+and+Clark%2C+P%2E+2018%2E+SciTaiL%3A+A+Textual+Entailment+Dataset+from+Science+Question+Answering%2E+In+AAAI%2C+volume+17%2C+41%E2%80%9342%2E)
11. Dagan, I.; Glickman, O.; and Magnini, B. 2006. The PASCAL Recognising Textual Entailment Challenge. In Quiñonero-Candela, J.; Dagan, I.; Magnini, B.; and d’Alché Buc, F., eds., Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, 177–190. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-540-33428-6. [find](https://scholar.google.com/scholar?q=Dagan%2C+I%2E%3B+Glickman%2C+O%2E%3B+and+Magnini%2C+B%2E+2006%2E+The+PASCAL+Recognising+Textual+Entailment+Challenge%2E+In+Qui%C3%B1onero%2DCandela%2C+J%2E%3B+Dagan%2C+I%2E%3B+Magnini%2C+B%2E%3B+and+d%E2%80%99Alch%C3%A9+Buc%2C+F%2E%2C+eds%2E%2C+Machine+Learning+Challe)
12. Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [↗](https://arxiv.org/abs/1412.6980)
13. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
14. Kuhn, T. S. 2012. The structure of scientific revolutions. University of Chicago press. [find](https://scholar.google.com/scholar?q=Kuhn%2C+T%2E+S%2E+2012%2E+The+structure+of+scientific+revolutions%2E+University+of+Chicago+press%2E)
15. Do, Q.; Chan, Y. S.; and Roth, D. 2011. Minimally Supervised Event Causality Identification. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, 294–303. Edinburgh, Scotland, UK.: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Do%2C+Q%2E%3B+Chan%2C+Y%2E+S%2E%3B+and+Roth%2C+D%2E+2011%2E+Minimally+Supervised+Event+Causality+Identification%2E+In+Proceedings+of+the+2011+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+294%E2%80%93303%2E+Edinbur)
16. Dunietz, J.; Levin, L.; and Carbonell, J. 2017. The BECauSE Corpus 2.0: Annotating Causality and Overlapping Relations. In Proceedings of the 11th Linguistic Annotation Workshop, 95–104. Valencia, Spain: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Dunietz%2C+J%2E%3B+Levin%2C+L%2E%3B+and+Carbonell%2C+J%2E+2017%2E+The+BECauSE+Corpus+2%2E0%3A+Annotating+Causality+and+Overlapping+Relations%2E+In+Proceedings+of+the+11th+Linguistic+Annotation+Workshop%2C+95%E2%80%93104%2E+Valencia%2C+Spa)
17. Gururangan, S.; Swayamdipta, S.; Levy, O.; Schwartz, R.; Bowman, S.; and Smith, N. A. 2018. Annotation Artifacts in Natural Language Inference Data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 107–112. [find](https://scholar.google.com/scholar?q=Gururangan%2C+S%2E%3B+Swayamdipta%2C+S%2E%3B+Levy%2C+O%2E%3B+Schwartz%2C+R%2E%3B+Bowman%2C+S%2E%3B+and+Smith%2C+N%2E+A%2E+2018%2E+Annotation+Artifacts+in+Natural+Language+Inference+Data%2E+In+Proceedings+of+the+2018+Conference+of+the+North+)
18. New Orleans, Klein, D.; and Manning, C. D. 2003. [Accurate Unlexicalized Parsing](https://md.research.elamur.ai/free/en/papers/klein-2003-accurate-unlexicalized-parsing/). In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 423–430. Sapporo, Japan: Association for Computational Linguistics.
19. Landis, J. R.; and Koch, G. G. 1977. The measurement of observer agreement for categorical data. biometrics, 159–174. [find](https://scholar.google.com/scholar?q=Landis%2C+J%2E+R%2E%3B+and+Koch%2C+G%2E+G%2E+1977%2E+The+measurement+of+observer+agreement+for+categorical+data%2E+biometrics%2C+159%E2%80%93174%2E)
20. Li, Z.; Ding, X.; Liu, T.; Hu, J. E.; and Van Durme, B. 2020. Guided Generation of Cause and Effect. In Bessiere, C., ed., Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, 3629–3636. International Joint Conferences on Artificial Intelligence Organization. Main track. [find](https://scholar.google.com/scholar?q=Li%2C+Z%2E%3B+Ding%2C+X%2E%3B+Liu%2C+T%2E%3B+Hu%2C+J%2E+E%2E%3B+and+Van+Durme%2C+B%2E+2020%2E+Guided+Generation+of+Cause+and+Effect%2E+In+Bessiere%2C+C%2E%2C+ed%2E%2C+Proceedings+of+the+Twenty%2DNinth+International+Joint+Conference+on+Artificial+)
21. Liu, X.; He, P.; Chen, W.; and Gao, J. 2019a. [Multi-Task Deep Neural Networks for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/liu-2019-multi-task-deep-neural-networks/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4487–4496. Florence, Italy: Association for Computational Linguistics.
22. Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
23. Luukkonen, T. 1992. Is scientists’ publishing behaviour rewardseeking? Scientometrics, 24: 297– 319. [↗](https://doi.org/10.1007/bf02017913)
24. Marelli, M.; Menini, S.; Baroni, M.; Bentivogli, L.; Bernardi, R.; and Zamparelli, R. 2014. [A SICK cure for the evaluation of compositional distributional semantic models](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/). In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), 216–223. Reykjavik, Iceland: European Language Resources Association (ELRA).
25. Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 1003–1011. Suntec, Singapore: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mintz%2C+M%2E%3B+Bills%2C+S%2E%3B+Snow%2C+R%2E%3B+and+Jurafsky%2C+D%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+ACL+an)
26. Nie, A.; Bennett, E.; and Goodman, N. 2019. Dis-Sent: Learning Sentence Representations from Explicit Discourse Relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4497–4510. Florence, Italy: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nie%2C+A%2E%3B+Bennett%2C+E%2E%3B+and+Goodman%2C+N%2E+2019%2E+Dis%2DSent%3A+Learning+Sentence+Representations+from+Explicit+Discourse+Relations%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computationa)
27. Nie, Y.; Williams, A.; Dinan, E.; Bansal, M.; Weston, J.; and Kiela, D. 2020. Adversarial NLI: A New Benchmark for Natural Language Understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4885–4901. Online: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Nie%2C+Y%2E%3B+Williams%2C+A%2E%3B+Dinan%2C+E%2E%3B+Bansal%2C+M%2E%3B+Weston%2C+J%2E%3B+and+Kiela%2C+D%2E+2020%2E+Adversarial+NLI%3A+A+New+Benchmark+for+Natural+Language+Understanding%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Asso)
28. Oh, J.-H.; Torisawa, K.; Hashimoto, C.; Sano, M.; De Saeger, S.; and Ohtake, K. 2013. Why-Question Answering using Intra- and Inter-Sentential Causal Relations. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1733–1743. Sofia, Bulgaria: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Oh%2C+J%2E%2DH%2E%3B+Torisawa%2C+K%2E%3B+Hashimoto%2C+C%2E%3B+Sano%2C+M%2E%3B+De+Saeger%2C+S%2E%3B+and+Ohtake%2C+K%2E+2013%2E+Why%2DQuestion+Answering+using+Intra%2D+and+Inter%2DSentential+Causal+Relations%2E+In+Proceedings+of+the+51st+Annual+Meeti)
29. Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 311–318.
30. Pennington, J.; Socher, R.; and Manning, C. D. 2014. [GloVe: Global Vectors for Word Representation](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/). In Empirical Methods in Natural Language Processing (EMNLP), 1532–1543.
31. Prasad, R.; Dinesh, N.; Lee, A.; Miltsakaki, E.; Robaldo, L.; Joshi, A.; and Webber, B. 2008. [The Penn Discourse TreeBank 2.0](https://md.research.elamur.ai/free/en/papers/prasad-2008-penn-discourse-treebank-2-0/). In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08). Marrakech, Morocco: European Language Resources Association (ELRA). [↗](https://doi.org/10.63317/55sr8v9kexr7)
32. Pruksachatkun, Y.; Phang, J.; Liu, H.; Htut, P. M.; Zhang, X.; Pang, R. Y.; Vania, C.; Kann, K.; and Bowman, S. R. 2020. [Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work?](https://md.research.elamur.ai/free/en/papers/pruksachatkun-2020-intermediate-task-transfer-learning-pretrained/) In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5231–5247. Online: Association for Computational Linguistics.
33. Radev, D. R.; Muthukrishnan, P.; and Qazvinian, V. 2009. The ACL Anthology Network. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries (NLPIR4DL), 54–61. Suntec City, Singapore: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Radev%2C+D%2E+R%2E%3B+Muthukrishnan%2C+P%2E%3B+and+Qazvinian%2C+V%2E+2009%2E+The+ACL+Anthology+Network%2E+In+Proceedings+of+the+2009+Workshop+on+Text+and+Citation+Analysis+for+Scholarly+Digital+Libraries+%28NLPIR4DL%29%2C+54%E2%80%9361%2E)
34. Radinsky, K.; Davidovich, S.; and Markovitch, S. 2012. Learning causality for news events prediction. In Proceedings of the 21st international conference on World Wide Web, 909–918. [find](https://scholar.google.com/scholar?q=Radinsky%2C+K%2E%3B+Davidovich%2C+S%2E%3B+and+Markovitch%2C+S%2E+2012%2E+Learning+causality+for+news+events+prediction%2E+In+Proceedings+of+the+21st+international+conference+on+World+Wide+Web%2C+909%E2%80%93918%2E)
35. Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems, 3266–3280.
36. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018. [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 353–355. Brussels, Belgium: Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1804.07461)
37. Webber, B.; Knott, A.; Stone, M.; and Joshi, A. 1999. [Discourse relations: A structural and presuppositional account using lexicalised TAG](https://md.research.elamur.ai/free/en/papers/webber-1999-discourse-relations-structural-presuppositional-account/). In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, 41–48.
38. Williams, A.; Nangia, N.; and Bowman, S. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 1112–1122. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Williams%2C+A%2E%3B+Nangia%2C+N%2E%3B+and+Bowman%2C+S%2E+2018%2E+A+Broad%2DCoverage+Challenge+Corpus+for+Sentence+Understanding+through+Inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+th)
39. Yang, Z.; Dai, Z.; Yang, Y.; Carbonell, J.; Salakhutdinov, R. R.; and Le, Q. V. 2019. [Xlnet: Generalized autoregressive pretraining for language understanding](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/). In Advances in neural information processing systems, 5753–5763. [↗](https://doi.org/10.48550/arxiv.1906.08237)
40. Zellers, R.; Bisk, Y.; Schwartz, R.; and Choi, Y. 2018. SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 93–104. Brussels, Belgium: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zellers%2C+R%2E%3B+Bisk%2C+Y%2E%3B+Schwartz%2C+R%2E%3B+and+Choi%2C+Y%2E+2018%2E+SWAG%3A+A+Large%2DScale+Adversarial+Dataset+for+Grounded+Commonsense+Inference%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natura)
41. Zhu, Y.; Kiros, R.; Zemel, R.; Salakhutdinov, R.; Urtasun, R.; Torralba, A.; and Fidler, S. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, 19–27.

*Updated 24 October 2026*