--- title: "ContractNLI: A Dataset for Document-level — Koreeda 2021" description: "Contract review is slow and expensive, and people who cannot afford lawyers often sign agreements without expert checking." authors: ["Yuta Koreeda", "Christopher Manning"] year: 2021 venue: "Findings of the Association for Computational Linguistics: EMNLP 2021" source: "https://aclanthology.org/2021.findings-emnlp.164/" doi: "10.18653/v1/2021.findings-emnlp.164" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/koreeda-2021-contractnli-dataset-document-level-natural/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/koreeda-2021-contractnli-dataset-document-level-natural/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) / ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts # ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts **Authors:** Yuta Koreeda, [Christopher Manning](https://md.research.elamur.ai/free/en/authors/christopher-manning/) · **Year:** 2021 · **Venue:** Findings of the Association for Computational Linguistics: EMNLP 2021 · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.findings-emnlp.164/) · [PDF](https://aclanthology.org/2021.findings-emnlp.164.pdf) · [DOI 10.18653/v1/2021.findings-emnlp.164](https://doi.org/10.18653/v1/2021.findings-emnlp.164) · [OpenAlex](https://openalex.org/W3201977280) **Citations:** 76 citations (openalex-snapshot · 2026-10-14) · [101](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Contract review is slow and expensive, and people who cannot afford lawyers often sign agreements without expert checking. This paper treats contract analysis as document-level natural language inference (NLI). Given a whole contract and a fixed list of hypotheses, a system must label each hypothesis as entailed, contradicted or not mentioned. It must also point to the supporting evidence, with each evidence unit being either a single sentence or one item from a list. The authors built ContractNLI, a corpus of 607 non-disclosure agreements (NDAs) collected from EDGAR and web search, converted to clean text and annotated against 17 hypotheses. Most documents far exceed BERT's input limit, and token-level answer extraction suits this task poorly. The authors therefore propose Span NLI BERT, which classifies marked spans with multiple labels and splits long contexts more carefully. Span NLI BERT clearly outperforms existing Transformer baselines. Still, the NLI decision proves harder than finding evidence, and rare labels and clauses negated by exceptions remain weak points. The work offers a realistic legal benchmark for reasoning over long documents. ## Key points - ContractNLI contains 607 annotated NDAs, which the authors describe as the largest annotated contract corpus at the time. It is split 70:10:20 into train, development and test sets (423 / 61 / 123), stratified by source format (plain text, HTML, PDF). - Each contract is judged against the same 17 hypotheses, such as survival of obligations or no reverse engineering. Labels are ENTAILMENT, CONTRADICTION and NOT MENTIONED, and evidence spans are marked only for the first two. - Training documents average about 2,254 tokens and 77.8 candidate spans. About 86% are longer than BERT's 512-token limit. - Span NLI BERT predicts evidence through multi-label classification over span markers instead of start/end tokens. With BERT base it even beats SQuAD-style BERT large. - Text was extracted with the authors' own PDF tool and segmented with Stanza. Annotation ran on Amazon Mechanical Turk at an average pay of 18.31 USD per hour. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 101 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 97 not yet. - [LCHAIM - Investigating Long Context Reasoning in Hebrew](https://md.research.elamur.ai/free/en/papers/findings-2025-malul-lchaim-investigating-long-context-reasoning/) - [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/guha-2023-legalbench-collaboratively-built-benchmark-measuring/) - [SCROLLS: Standardized CompaRison Over Long Language Sequences](https://md.research.elamur.ai/free/en/papers/shaham-2022-scrolls-standardized-comparison-over-long/) - [UL2: Unifying Language Learning Paradigms](https://md.research.elamur.ai/free/en/papers/tay-2022-ul2-unifying-language-learning-paradigms/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{koreeda2021contractnli, title = {{ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts}}, author = {Koreeda, Yuta and Manning, Christopher}, booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021}, year = {2021}, doi = {10.18653/v1/2021.findings-emnlp.164}, url = {https://doi.org/10.18653/v1/2021.findings-emnlp.164} } ``` - [← previous Contextualized Perturbation for Textual Adversarial Attack](https://md.research.elamur.ai/free/en/papers/li-2021-contextualized-perturbation-textual-adversarial-attack/) - [next → Contrastive Learning for Many-to-many Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/pan-2021-contrastive-learning-many-many-multilingual/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/2/) ## References 20 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT: The Muppets straight out of Law School. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2898– 2904. [find](https://scholar.google.com/scholar?q=Ilias+Chalkidis%2C+Manos+Fergadiotis%2C+Prodromos+Malakasiotis%2C+Nikolaos+Aletras%2C+and+Ion+Androutsopoulos%2E+2020%2E+LEGAL%2DBERT%3A+The+Muppets+straight+out+of+Law+School%2E+In+Findings+of+the+Association+for+Comp) [our card](https://md.research.elamur.ai/free/en/papers/chalkidis-2020-legal-bert-muppets-straight-out/) 2. Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Transactions on Intelligent Systems and Technology, 2(3). [find](https://scholar.google.com/scholar?q=Chih%2DChung+Chang+and+Chih%2DJen+Lin%2E+2011%2E+LIBSVM%3A+A+Library+for+Support+Vector+Machines%2E+ACM+Transactions+on+Intelligent+Systems+and+Technology%2C+2%283%29%2E) 3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N) 4. Exigent Group Limited. 2019. How GCs can thrive, not just survive. Technical report. [find](https://scholar.google.com/scholar?q=Exigent+Group+Limited%2E+2019%2E+How+GCs+can+thrive%2C+not+just+survive%2E+Technical+report%2E) 5. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. arXiv:2006.03654 [cs]. [find](https://scholar.google.com/scholar?q=Pengcheng+He%2C+Xiaodong+Liu%2C+Jianfeng+Gao%2C+and+Weizhu+Chen%2E+2021%2E+DeBERTa%3A+Decoding%2Denhanced+BERT+with+Disentangled+Attention%2E+arXiv%3A2006%2E03654+%5Bcs%5D%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2020-deberta-decoding-enhanced-bert-disentangled/) 6. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. arXiv. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks%2C+Collin+Burns%2C+Anya+Chen%2C+and+Spencer+Ball%2E+2021%2E+CUAD%3A+An+Expert%2DAnnotated+NLP+Dataset+for+Legal+Contract+Review%2E+arXiv%2E) [our card](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-cuad-expert-annotated-nlp-dataset/) 7. Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering. In Proceedings of the 2020 Natural Legal Language Processing (NLLP) Workshop. [find](https://scholar.google.com/scholar?q=Nils+Holzenberger%2C+Andrew+Blair%2DStanek%2C+and+Benjamin+Van+Durme%2E+2020%2E+A+Dataset+for+Statutory+Reasoning+in+Tax+Law+Entailment+and+Question+Answering%2E+In+Proceedings+of+the+2020+Natural+Legal+Language+) 8. Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3441–3460. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Yichen+Jiang%2C+Shikha+Bordia%2C+Zheng+Zhong%2C+Charles+Dognin%2C+Maneesh+Singh%2C+and+Mohit+Bansal%2E+2020%2E+HoVer%3A+A+Dataset+for+Many%2DHop+Fact+Extraction+And+Claim+Verification%2E+In+Findings+of+the+Association+fo) [our card](https://md.research.elamur.ai/free/en/papers/jiang-2020-hover-dataset-many-hop-fact/) 9. Yuta Koreeda and Christopher D. Manning. 2021. Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parser. arXiv. [find](https://scholar.google.com/scholar?q=Yuta+Koreeda+and+Christopher+D%2E+Manning%2E+2021%2E+Capturing+Logical+Structure+of+Visually+Structured+Documents+with+Multimodal+Transition+Parser%2E+arXiv%2E) 10. Language Technologies, Volume 1 (Long Papers), pages 809–819. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Language+Technologies%2C+Volume+1+%28Long+Papers%29%2C+pages+809%E2%80%93819%2E+Association+for+Computational+Linguistics%2E) 11. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+%C5%81uk) 12. Lucia Zheng, Neel Guha, Brandon R. Anderson, Peter Henderson, and Daniel E. Ho. 2021. When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset. In Proceedings of the 18th International Conference on Artificial Intelligence and Law. Association for Computing Mac… [find](https://scholar.google.com/scholar?q=Lucia+Zheng%2C+Neel+Guha%2C+Brandon+R%2E+Anderson%2C+Peter+Henderson%2C+and+Daniel+E%2E+Ho%2E+2021%2E+When+Does+Pretraining+Help%3F+Assessing+Self%2DSupervised+Learning+for+Law+and+the+CaseHOLD+Dataset%2E+In+Proceedings+of) 13. Spyretta Leivaditi, Julien Rossi, and Evangelos Kanoulas. 2020. A Benchmark for Lease Contract Review. arXiv. [find](https://scholar.google.com/scholar?q=Spyretta+Leivaditi%2C+Julien+Rossi%2C+and+Evangelos+Kanoulas%2E+2020%2E+A+Benchmark+for+Lease+Contract+Review%2E+arXiv%2E) 14. Marco Lippi, Przemysław Pałka, Giuseppe Contissa, Francesca Lagioia, Hans-Wolfgang Micklitz, Giovanni Sartor, and Paolo Torroni. 2019. CLAUDETTE: an automated detector of potentially unfair clauses in online terms of service. Artificial Intelligence and Law, 27(2):117–139. [find](https://scholar.google.com/scholar?q=Marco+Lippi%2C+Przemys%C5%82aw+Pa%C5%82ka%2C+Giuseppe+Contissa%2C+Francesca+Lagioia%2C+Hans%2DWolfgang+Micklitz%2C+Giovanni+Sartor%2C+and+Paolo+Torroni%2E+2019%2E+CLAUDETTE%3A+an+automated+detector+of+potentially+unfair+clauses+in) 15. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc) 16. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Peng+Qi%2C+Yuhao+Zhang%2C+Yuhui+Zhang%2C+Jason+Bolton%2C+and+Christopher+D%2E+Manning%2E+2020%2E+Stanza%3A+A+python+natural+language+processing+toolkit+for+many+human+languages%2E+In+Proceedings+of+the+58th+Annual+Meet) [our card](https://md.research.elamur.ai/free/en/papers/ud-2020-qi-stanza-python-natural-language-processing/) 17. Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu Yoshioka, Yoshinobu Kano, and Ken Satoh. 2020. [find](https://scholar.google.com/scholar?q=Juliano+Rabelo%2C+Mi%2DYoung+Kim%2C+Randy+Goebel%2C+Masaharu+Yoshioka%2C+Yoshinobu+Kano%2C+and+Ken+Satoh%2E+2020%2E) 18. COLIEE 2020: Methods for Legal Document Retrieval and Entailment. [find](https://scholar.google.com/scholar?q=COLIEE+2020%3A+Methods+for+Legal+Document+Retrieval+and+Entailment%2E) 19. The U.S. Securities and Exchange Commission. 2018. EDGAR® Public Dissemination Service Technical Specification. [find](https://scholar.google.com/scholar?q=The+U%2ES%2E+Securities+and+Exchange+Commission%2E+2018%2E+EDGAR%C2%AE+Public+Dissemination+Service+Technical+Specification%2E) 20. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human [find](https://scholar.google.com/scholar?q=James+Thorne%2C+Andreas+Vlachos%2C+Christos+Christodoulopoulos%2C+and+Arpit+Mittal%2E+2018%2E+FEVER%3A+a+Large%2Dscale+Dataset+for+Fact+Extraction+and+VERification%2E+In+Proceedings+of+the+2018+Conference+of+the+Nort) [our card](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/)