---
title: "DocNLI: A Large-scale Dataset for Document-level — Yin 2021"
description: "Most NLI resources pair one short premise sentence with one short hypothesis."
authors: ["Wenpeng Yin", "Dragomir Radev", "Caiming Xiong"]
year: 2021
venue: "Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021"
source: "https://aclanthology.org/2021.findings-acl.435/"
doi: "10.18653/v1/2021.findings-acl.435"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/yin-2021-docnli-large-scale-dataset-document/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/yin-2021-docnli-large-scale-dataset-document/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/) / DocNLI: A Large-scale Dataset for Document-level Natural Language Inference

# DocNLI: A Large-scale Dataset for Document-level Natural Language Inference

**Authors:** [Wenpeng Yin](https://md.research.elamur.ai/free/en/authors/wenpeng-yin/), [Dragomir Radev](https://md.research.elamur.ai/free/en/authors/dragomir-radev/), [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/) · **Year:** 2021 · **Venue:** Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.findings-acl.435/) · [PDF](https://aclanthology.org/2021.findings-acl.435.pdf) · [DOI 10.18653/v1/2021.findings-acl.435](https://doi.org/10.18653/v1/2021.findings-acl.435) · [OpenAlex](https://openalex.org/W3173529047)

**Citations:** 58 citations (openalex-snapshot · 2026-10-14) · [63](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Most NLI resources pair one short premise sentence with one short hypothesis. Many downstream uses need more than that, such as checking whether a whole summary is faithful to its source or verifying a claim against a full article. The authors built DocNLI to support inference at document scale. They recast five existing benchmarks into a single binary entail / not-entail task:

- ANLI, an NLI dataset;
- SQuAD, via the QA2D conversion of question-answer pairs into statements;
- three summarization corpora: DUC2001, CNN/DailyMail and Curation.

In every case the premise spans several sentences, usually a natural document like a news story. Hypotheses range from one sentence to passages of hundreds of words. For the summarization data, negative hypotheses are altered "fake" versions of the reference summaries. The construction was designed to minimize annotation artifacts, and a hypothesis-only baseline struggles to exploit such shortcuts.

A RoBERTa-large model trained on DocNLI transfers well even without task-specific tuning. It outperforms models trained on ANLI or MNLI on document-level tasks (FEVER-binary, MCTest) and holds up on sentence-level benchmarks. Fine-tuning improves results further. The data, code and checkpoints are publicly released.

## Key points
- **Sources:** DocNLI merges ANLI, QA2D/SQuAD, DUC2001, CNN/DailyMail and Curation.
  - ANLI's "neutral" and "contradict" classes are collapsed into "not entail".
  - QA2D is re-split 50k/7,236/8,275 (train/dev/test).
  - DUC2001's 600 document-summary pairs are split 400/50/150.
- **Hypothesis length:** hypotheses go well beyond single sentences, with summary-based ones reaching hundreds of words. The authors present it as the first NLI dataset of this kind closely tied to end tasks.
- **Document-level transfer:**
  - On MCTest, the DocNLI-trained model beats the prior state of the art by a large margin, both with and without fine-tuning.
  - On FEVER-binary, it generalizes better than models trained on MNLI or ANLI.
- **Sentence-level transfer:** without fine-tuning, the model reaches 78.17 accuracy on SciTail, above fully supervised ESIM, De-Att and DGEM. After fine-tuning it comes close to the SciTail SOTA (DeBERTa).
- **Release:** resources are available at github.com/salesforce/DocNLI.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

63 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 55 not yet.

- [QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization](https://md.research.elamur.ai/free/en/papers/fabbri-2022-qafacteval-improved-qa-based-factual/)
- [LCHAIM - Investigating Long Context Reasoning in Hebrew](https://md.research.elamur.ai/free/en/papers/findings-2025-malul-lchaim-investigating-long-context-reasoning/)
- [TRUE: Re-evaluating Factual Consistency Evaluation](https://md.research.elamur.ai/free/en/papers/honovich-2022-true-re-evaluating-factual-consistency/)
- [大语言模型的幻觉问题研究综述](https://md.research.elamur.ai/free/en/papers/liu-2024-hallucinations-large-language/)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://md.research.elamur.ai/free/en/papers/tang-2024-minicheck-efficient-fact-checking-llms/)
- [UL2: Unifying Language Learning Paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://md.research.elamur.ai/free/en/papers/vu-2022-spot-better-frozen-model-adaptation/)
- [AlignScore: Evaluating Factual Consistency with A Unified Alignment Function](https://md.research.elamur.ai/free/en/papers/zha-2023-alignscore-evaluating-factual-consistency-unified/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{yin2021docnli,
  title = {{DocNLI: A Large-scale Dataset for Document-level Natural Language Inference}},
  author = {Yin, Wenpeng and Radev, Dragomir and Xiong, Caiming},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.435},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.435}
}
```

- [← previous Do UD Trees Match Mention Spans in Coreference Annotations?](https://md.research.elamur.ai/free/en/papers/coref-2021-popel-do-ud-trees-match-mention/)
- [next → Document-Level Event Argument Extraction by Conditional Generation](https://md.research.elamur.ai/free/en/papers/li-2021-document-level-event-argument-extraction/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/)

## References

20 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. [↗](https://arxiv.org/abs/1907.11692)
2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. CoRR, abs/2004.05150. [↗](https://arxiv.org/abs/2004.05150)
3. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Evaluating the factual consistency of abstractive text summarization. CoRR, abs/1910.12840. [↗](https://arxiv.org/abs/1910.12840)
4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP, pages 632–642. [↗](https://doi.org/10.48550/arxiv.1508.05326)
5. Ramesh Nallapati, Bowen Zhou, Cı́cero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In CoNLL, pages 280– 290. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+C%C4%B1%CC%81cero+Nogueira+dos+Santos%2C+C%CC%A7aglar+Gu%CC%88lc%CC%A7ehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+text+summarization+using+sequence%2Dto%2Dsequence+rnns+and+beyond%2E+In+CoNLL%2C+pages+280%E2%80%93+290)
6. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for natural language inference. In ACL, pages 1657– 1668. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+natural+language+inference%2E+In+ACL%2C+pages+1657%E2%80%93+1668%2E)
7. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019. Combining fact extraction and verification with neural semantic matching networks. In AAAI, pages 6859–6866. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Haonan+Chen%2C+and+Mohit+Bansal%2E+2019%2E+Combining+fact+extraction+and+verification+with+neural+semantic+matching+networks%2E+In+AAAI%2C+pages+6859%E2%80%936866%2E)
8. Curation. 2020. Curation corpus base. [find](https://scholar.google.com/scholar?q=Curation%2E+2020%2E+Curation+corpus+base%2E)
9. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In ACL. [find](https://scholar.google.com/scholar?q=Yixin+Nie%2C+Adina+Williams%2C+Emily+Dinan%2C+Mohit+Bansal%2C+Jason+Weston%2C+and+Douwe+Kiela%2E+2020%2E+Adversarial+NLI%3A+A+new+benchmark+for+natural+language+understanding%2E+In+ACL%2E)
10. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches - erratum. Nat. Lang. Eng., 16(1):105. [↗](https://doi.org/10.1017/s1351324909990209)
11. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment Ankur P. Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP, pages 2249–2255. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+PASCAL+recognising+textual+entailment+Ankur+P%2E+Parikh%2C+Oscar+Ta%CC%88ckstro%CC%88m%2C+Dipanjan+Das%2C+and+Jakob+Uszkoreit%2E+2016%2E+A+decomposable+attention+mo) [our card](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)
12. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis only baselines in natural language inference. In *SEM@NAACL-HLT , pages 180–191. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Adam+Poliak%2C+Jason+Naradowsky%2C+Aparajita+Haldar%2C+Rachel+Rudinger%2C+and+Benjamin+Van+Durme%2E+2018%2E+Hypothesis+only+baselines+in+natural+language+inference%2E+In+%2ASEM%40NAACL%2DHLT+%2C+pages+180%E2%80%93191%2E+Association+)
13. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP, pages 2383–2392. [↗](https://doi.org/10.48550/arxiv.1606.05250) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
14. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pages 193–203. [find](https://scholar.google.com/scholar?q=Matthew+Richardson%2C+Christopher+J%2E+C%2E+Burges%2C+and+Erin+Renshaw%2E+2013%2E+Mctest%3A+A+challenge+dataset+for+the+open%2Ddomain+machine+comprehension+of+text%2E+In+EMNLP%2C+pages+193%E2%80%93203%2E) [our card](https://md.research.elamur.ai/free/en/papers/missing-2013-richardson-mctest-challenge-dataset-open-domain/)
15. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT, pages 809–819. [find](https://scholar.google.com/scholar?q=James+Thorne%2C+Andreas+Vlachos%2C+Christos+Christodoulopoulos%2C+and+Arpit+Mittal%2E+2018%2E+FEVER%3A+a+large%2Dscale+dataset+for+fact+extraction+and+verification%2E+In+NAACL%2DHLT%2C+pages+809%E2%80%93819%2E) [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/)
16. Masatoshi Tsuchiya. 2018. Performance impact caused by hidden bias of training data for recognizing textual entailment. In LREC. [find](https://scholar.google.com/scholar?q=Masatoshi+Tsuchiya%2E+2018%2E+Performance+impact+caused+by+hidden+bias+of+training+data+for+recognizing+textual+entailment%2E+In+LREC%2E) [our card](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/)
17. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998–6008. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NeurIPS%2C+pages+5998%E2%80%936008%2E)
18. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, pages 1112–1122. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R%2E+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2DHLT%2C+pages+1112%E2%80%931122%2E)
19. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.08144. [↗](https://arxiv.org/abs/1609.08144)
20. Jianxing Yu, Zhengjun Zha, and Jian Yin. 2019. Inferential machine comprehension: Answering questions by recursively deducing the evidence chain from text. In Proceedings of ACL, pages 2241– 2251. [find](https://scholar.google.com/scholar?q=Jianxing+Yu%2C+Zhengjun+Zha%2C+and+Jian+Yin%2E+2019%2E+Inferential+machine+comprehension%3A+Answering+questions+by+recursively+deducing+the+evidence+chain+from+text%2E+In+Proceedings+of+ACL%2C+pages+2241%E2%80%93+2251%2E)