---
title: "DisSent: Learning Sentence Representations from — Nie 2019"
description: "Good general-purpose sentence embeddings usually require either huge unlabeled corpora (as with BERT) or expensive hand-labeled sentence-pair data such…"
authors: ["Allen Nie", "Erin Bennett", "Noah Goodman"]
year: 2019
venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/P19-1442/"
doi: "10.18653/v1/p19-1442"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/nie-2019-dissent-learning-sentence-representations-explicit/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/nie-2019-dissent-learning-sentence-representations-explicit/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/) / DisSent: Learning Sentence Representations from Explicit Discourse Relations

# DisSent: Learning Sentence Representations from Explicit Discourse Relations

**Authors:** Allen Nie, Erin Bennett, [Noah Goodman](https://md.research.elamur.ai/free/en/authors/noah-goodman/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/P19-1442/) · [PDF](https://aclanthology.org/P19-1442.pdf) · [DOI 10.18653/v1/p19-1442](https://doi.org/10.18653/v1/p19-1442) · [OpenAlex](https://openalex.org/W2952750383)

**Citations:** 88 citations (openalex-snapshot · 2026-10-14) · [46](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary

Good general-purpose sentence embeddings usually require either huge unlabeled corpora (as with BERT) or expensive hand-labeled sentence-pair data such as the entailment sets behind InferSent. Nie, Bennett and Goodman propose a middle path. Writers routinely signal how two ideas relate with connectives like *because*, *but* or *so*, and the authors use these naturally occurring discourse markers as free supervision. They apply the Stanford CoreNLP dependency parser and marker-specific dependency patterns to BookCorpus. This lets them automatically pull out pairs of sentence-like clauses in their correct conceptual order, even when the marker's surface position varies. A model then has to guess the connecting marker from the two clauses alone. Training a BiLSTM encoder on this DisSent task yields fixed embeddings that roughly match InferSent on SentEval and beat it on some tasks. Using the task as an intermediate fine-tuning step for BERT-base gives state-of-the-art results on implicit relation classification in the Penn Discourse Treebank (PDTB). The approach is cheap, needs no manual annotation, and could in principle carry over to other languages through universal dependencies.

## Key points

- **Corpus:** The dataset is built from BookCorpus and contains about 4.7M clause pairs (4,706,292) covering 15 markers. Markers were chosen if they occur in more than 1% of PDTB cases. *but* and *and* each account for roughly 22% of pairs. Experiments use three marker subsets: ALL, 5 and 8.
- **Data efficiency:** DisSent Books 5 uses about 3.2M examples and 63.5M tokens. This is far less text than SkipThought (about 800M tokens) or BERT pre-training (about 3300M tokens).
- **Encoder architecture:** The encoder is a bidirectional LSTM with global max pooling over time. The two sentence vectors are combined with their average, difference and element-wise product, then passed to a softmax classifier over markers.
- **BERT fine-tuning:** BERT-base (12-layer Transformer) is trained on DisSent using the [CLS] representation, then fine-tuned further on PDTB tasks.
- **Extraction quality:** Pairs were aligned against PDTB gold annotations to check quality. Precision differs by marker, being higher for *because* and *so* and lower for *and* and *still*. When an aligned match is found, the extracted pair is generally close to the gold pair.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/en/topics/discourse-analysis/)

## Cited by

Cited by

46 papers of our library cite it (of the 140884 whose bibliographies we read); 15 of them have a page here, 31 not yet.

- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://md.research.elamur.ai/free/en/papers/cohan-2020-specter-document-level-representation-learning/)
- [CVAE-based Re-anchoring for Implicit Discourse Relation Classification](https://md.research.elamur.ai/free/en/papers/dou-2021-cvae-based-re-anchoring-implicit/)
- [Comparing PTB and UD information for PDTB discourse connective identification](https://md.research.elamur.ai/free/en/papers/han-2020-comparing-ptb-ud-information-pdtb/)
- [Implicit Discourse Relation Classification: We Need to Talk about Evaluation](https://md.research.elamur.ai/free/en/papers/kim-2020-implicit-discourse-relation-classification-we/)
- [Assessing Discourse Relations in Language Generation from GPT-2](https://md.research.elamur.ai/free/en/papers/ko-2020-assessing-discourse-relations-language-generation/)
- [Top-down Discourse Parsing via Sequence Labelling](https://md.research.elamur.ai/free/en/papers/koto-2021-top-down-discourse-parsing-sequence/)
- [Let's be explicit about that: Distant supervision for implicit discourse relation classification via connective prediction](https://md.research.elamur.ai/free/en/papers/kurfal-2021-let-s-be-explicit-about/)
- [Weakly-Supervised Modeling of Contextualized Event Embedding for Discourse Relations](https://md.research.elamur.ai/free/en/papers/lee-2020-weakly-supervised-modeling-contextualized-event/)
- [TED-CDB: A Large-Scale Chinese Discourse Relation Dataset on TED Talks](https://md.research.elamur.ai/free/en/papers/long-2020-ted-cdb-large-scale-chinese/)
- ["I'd rather just go to bed": Understanding Indirect Answers](https://md.research.elamur.ai/free/en/papers/louis-2020-i-d-rather-just-go/)
- [An Empirical Study of Synthetic Data Generation for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/omura-2024-empirical-study-synthetic-data-generation/)
- [Pragmatic competence of pre-trained language models through the lens of discourse connectives](https://md.research.elamur.ai/free/en/papers/pandia-2021-pragmatic-competence-pre-trained-language/)
- [Comparison of methods for explicit discourse connective identification across various domains](https://md.research.elamur.ai/free/en/papers/scholman-2021-comparison-methods-explicit-discourse-connective/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
- [The DISRPT 2021 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification](https://md.research.elamur.ai/free/en/papers/zeldes-2021-disrpt-2021-shared-task-elementary/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{nie2019dissent,
  title = {{DisSent: Learning Sentence Representations from Explicit Discourse Relations}},
  author = {Nie, Allen and Bennett, Erin and Goodman, Noah},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1442},
  url = {https://doi.org/10.18653/v1/p19-1442}
}
```

- [← previous Disentangled Representation Learning for Non-Parallel Text Style Transfer](https://md.research.elamur.ai/free/en/papers/john-2019-disentangled-representation-learning-non-parallel/)
- [next → DisSim: A Discourse-Aware Syntactic Text Simplification Framework for English and German](https://md.research.elamur.ai/free/en/papers/niklaus-2019-dissim-discourse-aware-syntactic-text/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/2/)

## References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yacine Jernite, Samuel R. Bowman, and David Sontag. 2017. Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning. arXiv:1705.00557 [cs, stat]. ArXiv: 1705.00557. — [arXiv:1705.00557](https://arxiv.org/abs/1705.00557) [find](https://scholar.google.com/scholar?q=Yacine+Jernite%2C+Samuel+R%2E+Bowman%2C+and+David+Sontag%2E+2017%2E+Discourse%2DBased+Objectives+for+Fast+Unsupervised+Sentence+Representation+Learning%2E+arXiv%3A1705%2E00557+%5Bcs%2C+stat%5D%2E+ArXiv%3A+1705%2E00557%2E+%E2%80%94+%5BarXiv%3A17)
2. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Yingyu+Liang%2C+and+Tengyu+Ma%2E+2017%2E+A+simple+but+tough%2Dto%2Dbeat+baseline+for+sentence+embeddings%2E+International+Conference+on+Learning+Representations%2E)
3. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329–344. [find](https://scholar.google.com/scholar?q=Yangfeng+Ji+and+Jacob+Eisenstein%2E+2015%2E+One+vector+is+not+enough%3A+Entity%2Daugmented+distributed+semantics+for+discourse+relations%2E+Transactions+of+the+Association+for+Computational+Linguistics%2C+3%3A329%E2%80%933) [our card](https://md.research.elamur.ai/free/en/papers/ji-2015-one-vector-is-not-enough/)
4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
5. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+page)
6. Alistair Knott. 1996. A data-driven methodology for motivating a set of coherence relations. [find](https://scholar.google.com/scholar?q=Alistair+Knott%2E+1996%2E+A+data%2Ddriven+methodology+for+motivating+a+set+of+coherence+relations%2E)
7. Xiang Lin, Shafiq Joty, Prathyusha Jwalapuram, and Saiful Bari. 2019. A unified linear-time framework for sentence-level discourse parsing. [find](https://scholar.google.com/scholar?q=Xiang+Lin%2C+Shafiq+Joty%2C+Prathyusha+Jwalapuram%2C+and+Saiful+Bari%2E+2019%2E+A+unified+linear%2Dtime+framework+for+sentence%2Dlevel+discourse+parsing%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2019-unified-linear-time-framework-sentence/)
8. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 1-Volume 1, pages 343–351. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Ziheng+Lin%2C+Min%2DYen+Kan%2C+and+Hwee+Tou+Ng%2E+2009%2E+Recognizing+implicit+discourse+relations+in+the+penn+discourse+treebank%2E+In+Proceedings+of+the+2009+Conference+on+Empirical+Methods+in+Natural+Language+) [our card](https://md.research.elamur.ai/free/en/papers/lin-2009-recognizing-implicit-discourse-relations-penn/)
9. Eric Malmi, Daniele Pighin, Sebastian Krause, and Mikhail Kozhevnikov. 2018. Automatic prediction of discourse connectives. In Proceedings of the 11th Language Resources and Evaluation Conference, Miyazaki, Japan. European Language Resource Association. [find](https://scholar.google.com/scholar?q=Eric+Malmi%2C+Daniele+Pighin%2C+Sebastian+Krause%2C+and+Mikhail+Kozhevnikov%2E+2018%2E+Automatic+prediction+of+discourse+connectives%2E+In+Proceedings+of+the+11th+Language+Resources+and+Evaluation+Conference%2C+Miy) [our card](https://md.research.elamur.ai/free/en/papers/malmi-2018-automatic-prediction-discourse-connectives/)
10. William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281. [find](https://scholar.google.com/scholar?q=William+C+Mann+and+Sandra+A+Thompson%2E+1988%2E+Rhetorical+structure+theory%3A+Toward+a+functional+theory+of+text+organization%2E+Text%2DInterdisciplinary+Journal+for+the+Study+of+Discourse%2C+8%283%29%3A243%E2%80%93281%2E) [our card](https://md.research.elamur.ai/free/en/papers/enud-1988-mann-rhetorical-structure-theory-functional/)
11. Gary Patterson and Andrew Kehler. 2013. Predicting the presence of discourse connectives. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 914–923. [find](https://scholar.google.com/scholar?q=Gary+Patterson+and+Andrew+Kehler%2E+2013%2E+Predicting+the+presence+of+discourse+connectives%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+914%E2%80%93923%2E) [our card](https://md.research.elamur.ai/free/en/papers/patterson-2013-predicting-presence-discourse-connectives/)
12. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+the+2018+Conference+of+t)
13. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. [find](https://scholar.google.com/scholar?q=Jason+Phang%2C+Thibault+Fe%CC%81vry%2C+and+Samuel+R+Bowman%2E+2018%2E+Sentence+encoders+on+stilts%3A+Supplementary+training+on+intermediate+labeled%2Ddata+tasks%2E+arXiv+preprint+arXiv%3A1811%2E01088%2E) [our card](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
14. Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, Adversarial connective-and Eric Xing. 2017. exploiting networks for implicit discourse relation classification. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1006– 1017, Vanc… [find](https://scholar.google.com/scholar?q=Lianhui+Qin%2C+Zhisong+Zhang%2C+Hai+Zhao%2C+Zhiting+Hu%2C+Adversarial+connective%2Dand+Eric+Xing%2E+2017%2E+exploiting+networks+for+implicit+discourse+relation+classification%2E+In+Proceedings+of+the+55th+Annual+Meet)
15. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E)
16. Prasad Rashmi, Dinesh Nihkil, Lee Alan, Miltsakaki Eleni, Robaldo Livio, Joshi Aravind, Webber Bonnie, et al. 2008. The penn discourse treebank 2.0. In Lexical Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Prasad+Rashmi%2C+Dinesh+Nihkil%2C+Lee+Alan%2C+Miltsakaki+Eleni%2C+Robaldo+Livio%2C+Joshi+Aravind%2C+Webber+Bonnie%2C+et+al%2E+2008%2E+The+penn+discourse+treebank+2%2E0%2E+In+Lexical+Resources+and+Evaluation+Conference%2E) [our card](https://md.research.elamur.ai/free/en/papers/disc-2008-prasad-penn-discourse-treebank-2-0/)
17. Sebastian Schuster and Christopher D Manning. 2016. Enhanced english universal dependencies: An improved representation for natural language understanding tasks. In Lexical Resources and Evaluation Conference. [find](https://scholar.google.com/scholar?q=Sebastian+Schuster+and+Christopher+D+Manning%2E+2016%2E+Enhanced+english+universal+dependencies%3A+An+improved+representation+for+natural+language+understanding+tasks%2E+In+Lexical+Resources+and+Evaluation+Co) [our card](https://md.research.elamur.ai/free/en/papers/ud-2016-schuster-enhanced-english-universal-dependencies-improved/)
18. Caroline Sporleder and Alex Lascarides. 2008. Using automatically labelled examples to classify rhetorical relations: An assessment. Natural Language Engineering, 14(3):369–416. [find](https://scholar.google.com/scholar?q=Caroline+Sporleder+and+Alex+Lascarides%2E+2008%2E+Using+automatically+labelled+examples+to+classify+rhetorical+relations%3A+An+assessment%2E+Natural+Language+Engineering%2C+14%283%29%3A369%E2%80%93416%2E)
19. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sandeep+Subramanian%2C+Adam+Trischler%2C+Yoshua+Bengio%2C+and+Christopher+J+Pal%2E+2018%2E+Learning+general+purpose+distributed+sentence+representations+via+large+scale+multi%2Dtask+learning%2E+In+International+Con)
20. Bonnie Webber, Matthew Stone, Aravind Joshi, and Alistair Knott. 2003. Anaphora and discourse structure. Computational linguistics, 29(4):545–587. [find](https://scholar.google.com/scholar?q=Bonnie+Webber%2C+Matthew+Stone%2C+Aravind+Joshi%2C+and+Alistair+Knott%2E+2003%2E+Anaphora+and+discourse+structure%2E+Computational+linguistics%2C+29%284%29%3A545%E2%80%93587%2E) [our card](https://md.research.elamur.ai/free/en/papers/webber-2003-anaphora-discourse-structure/)
21. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
22. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67–78. [find](https://scholar.google.com/scholar?q=Peter+Young%2C+Alice+Lai%2C+Micah+Hodosh%2C+and+Julia+Hockenmaier%2E+2014%2E+From+image+descriptions+to+visual+denotations%3A+New+similarity+metrics+for+semantic+inference+over+event+descriptions%2E+Transactions+of)
23. Frances Yung, Kevin Duh, Taku Komura, and Yuji Matsumoto. 2017. A psycholinguistic model for the marking of discourse relations. Dialogue & Discourse, 8(1):106–131. [find](https://scholar.google.com/scholar?q=Frances+Yung%2C+Kevin+Duh%2C+Taku+Komura%2C+and+Yuji+Matsumoto%2E+2017%2E+A+psycholinguistic+model+for+the+marking+of+discourse+relations%2E+Dialogue+%26+Discourse%2C+8%281%29%3A106%E2%80%93131%2E)
24. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–… [find](https://scholar.google.com/scholar?q=Yukun+Zhu%2C+Ryan+Kiros%2C+Rich+Zemel%2C+Ruslan+Salakhutdinov%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Aligning+books+and+movies%3A+Towards+story%2Dlike+visual+explanations+by+watching+movies+) [our card](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/)