---
title: "Ranking Generated Summaries by Correctness — Falke 2019"
description: "Abstractive summarizers write fluent text, but they often state things the source article does not support."
authors: ["Tobias Falke", "Leonardo F. R. Ribeiro", "Prasetya Ajie Utama", "Ido Dagan", "Iryna Gurevych"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1213/"
doi: "10.18653/v1/p19-1213"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference

# Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference

**Authors:** [Tobias Falke](https://md.research.elamur.ai/free/en/authors/tobias-falke/), [Leonardo F. R. Ribeiro](https://md.research.elamur.ai/free/en/authors/leonardo-f-r-ribeiro/), [Prasetya Ajie Utama](https://md.research.elamur.ai/free/en/authors/prasetya-ajie-utama/), [Ido Dagan](https://md.research.elamur.ai/free/en/authors/ido-dagan/), [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Institutions:** [Amazon](https://md.research.elamur.ai/free/en/companies/amazon/) · [Bar-Ilan University](https://md.research.elamur.ai/free/en/universities/bar-ilan-university/) · [Technische Universität Darmstadt](https://md.research.elamur.ai/free/en/universities/technische-universitat-darmstadt/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1213/) · [PDF](https://aclanthology.org/P19-1213.pdf) · [DOI 10.18653/v1/p19-1213](https://doi.org/10.18653/v1/p19-1213) · [OpenAlex](https://openalex.org/W2951211142)

**Citations:** 260 citations (OpenAlex, 2026-10-14) · [184](https://md.research.elamur.ai/free/en/papers/falke-2019-ranking-generated-summaries-correctness-interesting/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Abstractive summarizers write fluent text, but they often state things the source article does not support. This paper measures how common such errors are in single-document summarization on CNN-DM and tests whether natural language inference (NLI) can catch them. The authors built a crowdsourcing protocol in which workers label each summary sentence against the article, with labels merged by MACE. Using it, they found that roughly a quarter of the outputs from the strongest systems (FAS, BUS) contain errors.

They then reranked beam-search alternatives by entailment. Each candidate is scored by taking every summary sentence's best entailment probability against any source sentence and averaging those values. Five NLI models were compared. Off-the-shelf models gave small or inconsistent gains. ESIM even increased errors on the test set, and only BERT helped substantially. A simpler test of choosing between correct and incorrect sentence pairs showed a large gap to human accuracy.

The main lesson is that strong benchmark NLI scores do not carry over to this task. The released annotations offer a new extrinsic test set for NLI.

## Key points
- **Error rates on 100 CNN-DM test summaries (9 crowdworkers each):** PGC 8%, FAS 26%, BUS 25%. Correctness did not track ROUGE, and the more abstractive systems made more errors.
- **Crowdsourcing reliability:** agreement with an expert gold standard (50 summaries, 140 sentences) became substantial with 3 workers and plateaued at 9, with Cohen's κ of 0.74.
- **NLI models tested:** DA and ESIM (AllenNLP, with ELMo, trained on SNLI), InferSent, SSE, and BERT-base fine-tuned on MultiNLI.
- **Reranking results:** on 200 validation documents (5 FAS hypotheses each), the beam-order baseline had 42.1% incorrect summaries. BERT gave the clearest gain but still left over half of the errors. On the test set, ESIM reranking raised errors from 26% to 29%.
- **Sentence-pair test (373 triples):** the share of pairs where the incorrect sentence was preferred was 50% for random choice, 32.4% for ESIM (best model), 35.9% for BERT, and 16.1% for humans. Likely causes are domain shift and reliance on lexical overlap. The data are released via TUdatalib.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

Topics: [Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{falke2019ranking,
  title = {{Ranking Generated Summaries by Correctness: An Interesting but Challenging Application for Natural Language Inference}},
  author = {Falke, Tobias and Ribeiro, Leonardo F. R. and Utama, Prasetya Ajie and Dagan, Ido and Gurevych, Iryna},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1213},
  url = {https://doi.org/10.18653/v1/p19-1213}
}
```

- [← previous OpenDialKG: Explainable Conversational Reasoning with Attention-based Walks over Knowledge Graphs](https://md.research.elamur.ai/free/en/papers/moon-2019-opendialkg-explainable-conversational-reasoning-attention/)
- [next → FreeLB: Enhanced Adversarial Training for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/zhu-2019-freelb-enhanced-adversarial-training-natural/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

27 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A Large Annotated Corpus for Learning Natural Language Inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+Large+Annotated+Corpus+for+Learning+Natural+Language+Inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
2. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. Faithful to the Original: Fact Aware Neural Abstractive Summarization. In Proceedings of the Thiry-Second AAAI Conference on Artificial Intelligence, pages 4784–4791, New Orleans, LA, USA. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+Wenjie+Li%2C+and+Sujian+Li%2E+2018%2E+Faithful+to+the+Original%3A+Fact+Aware+Neural+Abstractive+Summarization%2E+In+Proceedings+of+the+Thiry%2DSecond+AAAI+Conference+on+Artificial+Intellige)
3. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced LSTM for Natural Language Inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1657–1668, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiaodan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+LSTM+for+Natural+Language+Inference%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computa)
4. Yen-Chun Chen and Mohit Bansal. 2018. Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 675–686, Melbourne, Australia. [find](https://scholar.google.com/scholar?q=Yen%2DChun+Chen+and+Mohit+Bansal%2E+2018%2E+Fast+Abstractive+Summarization+with+Reinforce%2DSelected+Sentence+Rewriting%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguist) [our card](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/)
5. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive Sentence Summarization with Attentive Recurrent Neural Networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93– 98, San D… [find](https://scholar.google.com/scholar?q=Sumit+Chopra%2C+Michael+Auli%2C+and+Alexander+M%2E+Rush%2E+2016%2E+Abstractive+Sentence+Summarization+with+Attentive+Recurrent+Neural+Networks%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapte) [our card](https://md.research.elamur.ai/free/en/papers/chopra-2016-abstractive-sentence-summarization-attentive-recurrent/)
6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+Learning+of+Universal+Sentence+Representations+from+Natural+Language+Inference+Data%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
7. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment, pages 177–190, Berlin, Heidelberg. Springer Berlin Heidelber… [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2006%2E+The+PASCAL+Recognising+Textual+Entailment+Challenge%2E+In+Machine+Learning+Challenges%2E+Evaluating+Predictive+Uncertainty%2C+Visual+Object+Classificati)
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+Deep+Bidirectional+Transformers+for+Language+Understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+N)
9. Lisa Fan, Dong Yu, and Lu Wang. 2018. Robust Neural Abstractive Summarization Systems and Evaluation against Adversarial Information. In NIPS 2018 Interpretability and Robustness for Audio, Speech and Language Workshop, Montreal, Canada. [find](https://scholar.google.com/scholar?q=Lisa+Fan%2C+Dong+Yu%2C+and+Lu+Wang%2E+2018%2E+Robust+Neural+Abstractive+Summarization+Systems+and+Evaluation+against+Adversarial+Information%2E+In+NIPS+2018+Interpretability+and+Robustness+for+Audio%2C+Speech+and)
10. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-Up Abstractive Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109, Brussels, Belgium. [find](https://scholar.google.com/scholar?q=Sebastian+Gehrmann%2C+Yuntian+Deng%2C+and+Alexander+Rush%2E+2018%2E+Bottom%2DUp+Abstractive+Summarization%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+4098%E2%80%934) [our card](https://md.research.elamur.ai/free/en/papers/gehrmann-2018-bottom-up-abstractive-summarization/)
11. Han Guo, Ramakanth Pasunuru, and Mohit Bansal. 2018. Soft Layer-Specific Multi-Task Summarization with Entailment and Question Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 687– 697, Melbourne, Australia. [find](https://scholar.google.com/scholar?q=Han+Guo%2C+Ramakanth+Pasunuru%2C+and+Mohit+Bansal%2E+2018%2E+Soft+Layer%2DSpecific+Multi%2DTask+Summarization+with+Entailment+and+Question+Generation%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+)
12. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems, pages 1693–1701, Montreal, Canada. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Toma%CC%81s%CC%8C+Koc%CC%8Cisky%CC%81%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+Machines+to+Read+and+Comprehend%2E+In+Proceedings+of+the+28th+In) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
13. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning Whom to Trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130, Atlanta, GA, USA. [find](https://scholar.google.com/scholar?q=Dirk+Hovy%2C+Taylor+Berg%2DKirkpatrick%2C+Ashish+Vaswani%2C+and+Eduard+Hovy%2E+2013%2E+Learning+Whom+to+Trust+with+MACE%2E+In+Proceedings+of+the+2013+Conference+of+the+North+American+Chapter+of+the+Association+for+) [our card](https://md.research.elamur.ai/free/en/papers/hovy-2013-learning-whom-trust-mace/)
14. Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. Ensure the Correctness of the Summary: Incorporate Entailment Knowledge into Abstractive Sentence Summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1430–1441, Santa Fe, NM, USA. [find](https://scholar.google.com/scholar?q=Haoran+Li%2C+Junnan+Zhu%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+2018%2E+Ensure+the+Correctness+of+the+Summary%3A+Incorporate+Entailment+Knowledge+into+Abstractive+Sentence+Summarization%2E+In+Proceedings+of+the+27)
15. Jiwei Li, Will Monroe, and Dan Jurafsky. 2016. A Simple, Fast Diverse Decoding Algorithm for Neural Generation. ArXiv preprint 1611.08562. — [arXiv:1611.08562](https://arxiv.org/abs/1611.08562) [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Dan+Jurafsky%2E+2016%2E+A+Simple%2C+Fast+Diverse+Decoding+Algorithm+for+Neural+Generation%2E+ArXiv+preprint+1611%2E08562%2E+%E2%80%94+%5BarXiv%3A1611%2E08562%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1611%2E08562%29)
16. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. [find](https://scholar.google.com/scholar?q=R%2E+Thomas+McCoy%2C+Ellie+Pavlick%2C+and+Tal+Linzen%2E+2019%2E+Right+for+the+Wrong+Reasons%3A+Diagnosing+Syntactic+Heuristics+in+Natural+Language+Inference%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Assoc) [our card](https://md.research.elamur.ai/free/en/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
17. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280–290, Berlin, Germany. [find](https://scholar.google.com/scholar?q=Ramesh+Nallapati%2C+Bowen+Zhou%2C+Cicero+dos+Santos%2C+Caglar+Gulcehre%2C+and+Bing+Xiang%2E+2016%2E+Abstractive+Text+Summarization+using+Sequence%2Dto%2Dsequence+RNNs+and+Beyond%2E+In+Proceedings+of+The+20th+SIGNLL+Con)
18. Yixin Nie and Mohit Bansal. Shortcut-Stacked Sentence Encoders for Multi-Domain Inference. In Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, pages 41–45, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Yixin+Nie+and+Mohit+Bansal%2E+Shortcut%2DStacked+Sentence+Encoders+for+Multi%2DDomain+Inference%2E+In+Proceedings+of+the+2nd+Workshop+on+Evaluating+Vector+Space+Representations+for+NLP%2C+pages+41%E2%80%9345%2C+Copenhage)
19. Nangia Nikita and Samual R. Bowman. 2019. Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. [find](https://scholar.google.com/scholar?q=Nangia+Nikita+and+Samual+R%2E+Bowman%2E+2019%2E+Human+vs%2E+Muppet%3A+A+Conservative+Estimate+of+Human+Performance+on+the+GLUE+Benchmark%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computa)
20. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A Decomposable Attention Model for Natural Language Inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2249–2255, Austin, Texas. [find](https://scholar.google.com/scholar?q=Ankur+Parikh%2C+Oscar+Ta%CC%88ckstro%CC%88m%2C+Dipanjan+Das%2C+and+Jakob+Uszkoreit%2E+2016%2E+A+Decomposable+Attention+Model+for+Natural+Language+Inference%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+N) [our card](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)
21. Ramakanth Pasunuru and Mohit Bansal. 2018. Multi-Reward Reinforced Summarization with Saliency and Entailment. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 646–653, New Orleans, LA, USA. [find](https://scholar.google.com/scholar?q=Ramakanth+Pasunuru+and+Mohit+Bansal%2E+2018%2E+Multi%2DReward+Reinforced+Summarization+with+Saliency+and+Entailment%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+Association+fo) [our card](https://md.research.elamur.ai/free/en/papers/pasunuru-2018-multi-reward-reinforced-summarization-saliency/)
22. Ramakanth Pasunuru, Han Guo, and Mohit Bansal. 2017. Towards Improving Abstractive Summarization via Entailment Generation. In Proceedings of the Workshop on New Frontiers in Summarization, pages 27–32, Copenhagen, Denmark. [find](https://scholar.google.com/scholar?q=Ramakanth+Pasunuru%2C+Han+Guo%2C+and+Mohit+Bansal%2E+2017%2E+Towards+Improving+Abstractive+Summarization+via+Entailment+Generation%2E+In+Proceedings+of+the+Workshop+on+New+Frontiers+in+Summarization%2C+pages+27%E2%80%933)
23. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+Contextualized+Word+Representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
24. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+Neural+Attention+Model+for+Abstractive+Sentence+Summarization%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language)
25. Marta Sabou, Kalina Bontcheva, Leon Derczynski, and Arno Scharl. 2014. Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. In Proceedings of the Ninth International Conference on Language Resources and Evaluation, Reykjavik, Iceland. [find](https://scholar.google.com/scholar?q=Marta+Sabou%2C+Kalina+Bontcheva%2C+Leon+Derczynski%2C+and+Arno+Scharl%2E+2014%2E+Corpus+Annotation+through+Crowdsourcing%3A+Towards+Best+Practice+Guidelines%2E+In+Proceedings+of+the+Ninth+International+Conference+o)
26. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1073–1083, Vancouver, Canada. [find](https://scholar.google.com/scholar?q=Abigail+See%2C+Peter+J%2E+Liu%2C+and+Christopher+D%2E+Manning%2E+2017%2E+Get+To+The+Point%3A+Summarization+with+Pointer%2DGenerator+Networks%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
27. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1112–1122,… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+Broad%2DCoverage+Challenge+Corpus+for+Sentence+Understanding+through+Inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)