---
title: "Pragmatic competence of pre-trained language — Pandia 2021"
description: "How much pragmatic knowledge do masked language models acquire from pre-training alone? The authors probe this through discourse connectives, words like…"
authors: ["Lalchand Pandia", "Yan Cong", "Allyson Ettinger"]
year: 2021
venue: "Proceedings of the 25th Conference on Computational Natural Language Learning"
source: "https://aclanthology.org/2021.conll-1.29/"
doi: "10.18653/v1/2021.conll-1.29"
arxiv: "2109.12951"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/papers/pandia-2021-pragmatic-competence-pre-trained-language/"
---
[Elamur Research](https://md.research.elamur.ai/) · [this page in HTML](https://research.elamur.ai/free/papers/pandia-2021-pragmatic-competence-pre-trained-language/)

[home](https://md.research.elamur.ai/) / [papers](https://md.research.elamur.ai/free/papers/) / [2021](https://md.research.elamur.ai/free/papers/2021/page/3/) / Pragmatic competence of pre-trained language models through the lens of discourse connectives

# Pragmatic competence of pre-trained language models through the lens of discourse connectives

**Authors:** Lalchand Pandia, Yan Cong, [Allyson Ettinger](https://md.research.elamur.ai/free/authors/allyson-ettinger/) · **Year:** 2021 · **Venue:** Proceedings of the 25th Conference on Computational Natural Language Learning · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.conll-1.29/) · [PDF](https://aclanthology.org/2021.conll-1.29.pdf) · [DOI 10.18653/v1/2021.conll-1.29](https://doi.org/10.18653/v1/2021.conll-1.29) · [arXiv 2109.12951](https://arxiv.org/abs/2109.12951) · [OpenAlex](https://openalex.org/W3210694367)

**Citations:** 21 citations (openalex-snapshot · 2026-10-14) · [21](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
How much pragmatic knowledge do masked language models acquire from pre-training alone? The authors probe this through discourse connectives, words like "but" or "so" that mark how two events relate and can carry implied meanings. They use no fine-tuning. Every test is a cloze task: the connective is masked, and the probabilities the model assigns to candidate connectives are compared. A broad first experiment uses explicit single-word connectives from the Penn Discourse Treebank (PDTB-2). Follow-up tests adapt controlled items from psycholinguistic studies. These check whether models can tell causal from concessive relations in near-identical minimal pairs, handle temporal implicatures when they are reinforced or cancelled, and favour connectives that keep event mentions in chronological order. On natural text, models pick the correct connective roughly half the time. On the controlled items their sensitivity drops sharply, with some models close to 0%, and humanlike temporal preferences are weak. The authors conclude that word-prediction pre-training does not by itself produce real pragmatic competence. Decent scores on natural data may instead reflect reliance on shallow lexical and syntactic cues.

## Key points
- Eight masked LMs were tested via the Wolf et al. (2020) implementation:
  - BERT BASE and BERT LARGE (uncased)
  - RoBERTa BASE and RoBERTa LARGE
  - ALBERT v2 BASE, LARGE, XLARGE and XXLARGE
- The PDTB-2 experiment used 17,476 items with explicit single-word connectives. A prediction counted as correct if the gold connective outranked the other 65 candidates (66 in total).
- Overall PDTB accuracy ranged from 0.42 (ALBERT BASE) to 0.66 (RoBERTa LARGE). Accuracy generally rose with model size within each family.
- Accuracy varied strongly by relation type:
  - Expansion.Conjunction reached up to 0.79.
  - Causal Result stayed between 0.09 and 0.34.
  - This pattern suggests models lean on versatile default connectives.
- In the event-order test, the share of item pairs showing the humanlike chronological preference never exceeded 41%, against a 25% chance level. Code and test data are released on GitHub (lalchand-pandia/Pragmatic-competence-in-PLM).

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/shelves/linguistics/)

Topics: [Discourse analysis](https://md.research.elamur.ai/free/topics/discourse-analysis/)

## Cited by

Cited by

21 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 20 not yet.

- [Is It JUST Semantics? A Case Study of Discourse Particle Understanding in LLMs](https://md.research.elamur.ai/free/papers/sheffield-2025-is-it-textit-just-semantics/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{pandia2021pragmatic,
  title = {{Pragmatic competence of pre-trained language models through the lens of discourse connectives}},
  author = {Pandia, Lalchand and Cong, Yan and Ettinger, Allyson},
  booktitle = {Proceedings of the 25th Conference on Computational Natural Language Learning},
  year = {2021},
  doi = {10.18653/v1/2021.conll-1.29},
  eprint = {2109.12951},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.conll-1.29}
}
```

- [← previous Points of Contact: The Shared Intellectual History of Vocalisation in Syriac, Arabic, and Hebrew](https://md.research.elamur.ai/free/papers/hbobook-obp0271-posegay-points-contact/)
- [next → Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://md.research.elamur.ai/free/papers/liu-2021-pre-train-prompt-predict-systematic/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/papers/2021/page/3/)

## References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Robyn Carston. 1988. Implicature, explicature, and truth-theoretic semantics. In Ruth M. Kempson, editor, Mental representations: The interface between language and reality, pages 155–181. Cambridge Univ. Press. [find](https://scholar.google.com/scholar?q=Robyn+Carston%2E+1988%2E+Implicature%2C+explicature%2C+and+truth%2Dtheoretic+semantics%2E+In+Ruth+M%2E+Kempson%2C+editor%2C+Mental+representations%3A+The+interface+between+language+and+reality%2C+pages+155%E2%80%93181%2E+Cambridge+U)
2. Gennaro Chierchia, Danny Fox, and Benjamin Spector. 2012. Scalar implicature as a grammatical phenomenon. In Semantics, volume 3, pages 2297– 2331. Mouton de Gruyter. [find](https://scholar.google.com/scholar?q=Gennaro+Chierchia%2C+Danny+Fox%2C+and+Benjamin+Spector%2E+2012%2E+Scalar+implicature+as+a+grammatical+phenomenon%2E+In+Semantics%2C+volume+3%2C+pages+2297%E2%80%93+2331%2E+Mouton+de+Gruyter%2E)
3. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [↗](https://arxiv.org/abs/1907.11692)
4. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448. [find](https://scholar.google.com/scholar?q=Tom+McCoy%2C+Ellie+Pavlick%2C+and+Tal+Linzen%2E+2019%2E+Right+for+the+wrong+reasons%3A+Diagnosing+syntactic+heuristics+in+natural+language+inference%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association) [our card](https://md.research.elamur.ai/free/papers/mccoy-2019-right-wrong-reasons-diagnosing-syntactic/)
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT (1). [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2DHLT+%281%29%2E)
6. Allen Nie, Erin Bennett, and Noah Goodman. 2019. DisSent: Learning sentence representations from explicit discourse relations. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4497–4510, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Allen+Nie%2C+Erin+Bennett%2C+and+Noah+Goodman%2E+2019%2E+DisSent%3A+Learning+sentence+representations+from+explicit+discourse+relations%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computat) [our card](https://md.research.elamur.ai/free/papers/nie-2019-dissent-learning-sentence-representations-explicit/)
7. H. Drenhaus, V. Demberg, J. Koehne, and F. Delogu. 2014. Incremental and predictive discourse processing based on causal and concessive discourse markers: Erp studies on german and english. In Proceedings of the Annual Meeting of the Cognitive Science Society, 36. [find](https://scholar.google.com/scholar?q=H%2E+Drenhaus%2C+V%2E+Demberg%2C+J%2E+Koehne%2C+and+F%2E+Delogu%2E+2014%2E+Incremental+and+predictive+discourse+processing+based+on+causal+and+concessive+discourse+markers%3A+Erp+studies+on+german+and+english%2E+In+Proceed)
8. I Noveck and Florelle Chevaux. 2002. The pragmatic development of and. In BUCLD Proceedings, volume 26, pages 453–463. [find](https://scholar.google.com/scholar?q=I+Noveck+and+Florelle+Chevaux%2E+2002%2E+The+pragmatic+development+of+and%2E+In+BUCLD+Proceedings%2C+volume+26%2C+pages+453%E2%80%93463%2E)
9. Danny Fox. 2007. Free choice and the theory of scalar implicatures. In Uli Sauerland and Penka Stateva, editors, Presupposition and Implicature in Compositional Semantics, pages 71–120. Palgrave Macmillan. [↗](https://doi.org/10.1057/9780230210752_4)
10. Gary Patterson and Andrew Kehler. 2013. Predicting the presence of discourse connectives. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 914–923, Seattle, Washington, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Gary+Patterson+and+Andrew+Kehler%2E+2013%2E+Predicting+the+presence+of+discourse+connectives%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+914%E2%80%93923%2C+Seat) [our card](https://md.research.elamur.ai/free/papers/patterson-2013-predicting-presence-discourse-connectives/)
11. Emily Pitler and A. Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In ACL. [find](https://scholar.google.com/scholar?q=Emily+Pitler+and+A%2E+Nenkova%2E+2009%2E+Using+syntax+to+disambiguate+explicit+discourse+connectives+in+text%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/papers/pitler-2009-syntax-disambiguate-explicit-discourse-connectives/)
12. B. Geurts. 2010. Quantity Implicatures. Cambridge University Press. [find](https://scholar.google.com/scholar?q=B%2E+Geurts%2E+2010%2E+Quantity+Implicatures%2E+Cambridge+University+Press%2E)
13. H. P. Grice. 1989. Studies in the Way of Words. ACLS Humanities E-Book. Harvard University Press. [find](https://scholar.google.com/scholar?q=H%2E+P%2E+Grice%2E+1989%2E+Studies+in+the+Way+of+Words%2E+ACLS+Humanities+E%2DBook%2E+Harvard+University+Press%2E)
14. Emily Pitler, Mridhula Raghupathy, Hena Mehta, A. Nenkova, Alan Lee, and A. Joshi. 2008. Easily identifiable discourse relations. In COLING. [find](https://scholar.google.com/scholar?q=Emily+Pitler%2C+Mridhula+Raghupathy%2C+Hena+Mehta%2C+A%2E+Nenkova%2C+Alan+Lee%2C+and+A%2E+Joshi%2E+2008%2E+Easily+identifiable+discourse+relations%2E+In+COLING%2E) [our card](https://md.research.elamur.ai/free/papers/pitler-2008-easily-identifiable-discourse-relations/)
15. H.P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Syntax and Semantics, volume 3, pages 41–58. Academic Press, New York. [↗](https://doi.org/10.1163/9789004368811_003)
16. Roni Katzir. 2007. Structurally-defined alternatives. Linguistics and Philosophy, 30(6):669–690. [↗](https://doi.org/10.1007/s10988-008-9029-y)
17. Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Discourse probing of pretrained language models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3849–3864, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Fajri+Koto%2C+Jey+Han+Lau%2C+and+Timothy+Baldwin%2E+2021%2E+Discourse+probing+of+pretrained+language+models%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Computat)
18. Murathan Kurfalı and Robert Östling. 2021. Probing multilingual language models for discourse. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021), pages 8–19, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48448/7ahp-rt73)
19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. ICLR 2020. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+ALBERT%3A+A+lite+BERT+for+self%2Dsupervised+learning+of+language+representations%2E+ICLR+2020%2E)
20. Stephen Politzer-Ahles, Ming Xiang, and Diogo Almeida. 2017. "before" and "after": Investigating the relationship between temporal connectives and chronological ordering using event-related potentials. PloS one, 12(4):e0175199. [↗](https://doi.org/10.1371/journal.pone.0175199)
21. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Milt-sakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). [↗](https://doi.org/10.63317/55sr8v9kexr7) [our card](https://md.research.elamur.ai/free/papers/disc-2008-prasad-penn-discourse-treebank-2-0/)
22. J. Rett. 2014. The Semantics of Evaluativity. Oxford Studies in Theoretical Linguistics. OUP Oxford. [↗](https://doi.org/10.1093/acprof:oso/9780199602476.001.0001)
23. J.M. Sadock. 1978. Pragmatics. In P. Cole, editor, Presupposition and Implicature in Compositional Semantics, volume 9, pages 281––298. Academic Press. [find](https://scholar.google.com/scholar?q=J%2EM%2E+Sadock%2E+1978%2E+Pragmatics%2E+In+P%2E+Cole%2C+editor%2C+Presupposition+and+Implicature+in+Compositional+Semantics%2C+volume+9%2C+pages+281%E2%80%93%E2%80%93298%2E+Academic+Press%2E)
24. Uli Sauerland. 2004. Scalar implicatures in complex sentences. Linguistics and Philosophy, 27(3):367– 391. [↗](https://doi.org/10.1023/b:ling.0000023378.71748.db)
25. Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019. Mining discourse markers for unsupervised sentence representation learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3477–3486, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.1903.11850) [our card](https://md.research.elamur.ai/free/papers/sileo-2019-mining-discourse-markers-unsupervised-sentence/)
26. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. arXiv preprint arXiv:2104.06644. [↗](https://arxiv.org/abs/2104.06644) [our card](https://md.research.elamur.ai/free/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/)
27. Bonnie Webber, Rashmi Prasad, and Alan Lee. 2019. Ambiguity in explicit discourse connectives. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 134– 141, Gothenburg, Sweden. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Bonnie+Webber%2C+Rashmi+Prasad%2C+and+Alan+Lee%2E+2019%2E+Ambiguity+in+explicit+discourse+connectives%2E+In+Proceedings+of+the+13th+International+Conference+on+Computational+Semantics+%2D+Long+Papers%2C+pages+134%E2%80%93+) [our card](https://md.research.elamur.ai/free/papers/webber-2019-ambiguity-explicit-discourse-connectives/)
28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6) [our card](https://md.research.elamur.ai/free/papers/wolf-2020-transformers-state-art-natural-language/)