---
title: "Pragmatic inference of scalar implicature by LLMs — Cho 2024"
description: "This student workshop paper asks whether early transformer LLMs read the quantifier some literally (\"at least one, maybe all\") or pragmatically (\"not…"
authors: ["Ye-eun Cho", "Seong mook Kim"]
year: 2024
venue: "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)"
source: "https://aclanthology.org/2024.acl-srw.2/"
doi: "10.18653/v1/2024.acl-srw.2"
arxiv: "2408.06673"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/cho-2024-pragmatic-inference-scalar-implicature-llms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/cho-2024-pragmatic-inference-scalar-implicature-llms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/) / Pragmatic inference of scalar implicature by LLMs

# Pragmatic inference of scalar implicature by LLMs

**Authors:** Ye-eun Cho, Seong mook Kim · **Year:** 2024 · **Venue:** Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.acl-srw.2/) · [PDF](https://aclanthology.org/2024.acl-srw.2.pdf) · [DOI 10.18653/v1/2024.acl-srw.2](https://doi.org/10.18653/v1/2024.acl-srw.2) · [arXiv 2408.06673](https://arxiv.org/abs/2408.06673) · [OpenAlex](https://openalex.org/W4402671408)

**Citations:** 3 citations (openalex-snapshot · 2026-10-14) · [13](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
This student workshop paper asks whether early transformer LLMs read the quantifier *some* literally ("at least one, maybe all") or pragmatically ("not all"). It also asks whether conversational context shifts that reading. The authors pulled sentences of the form 'some + NP' from the British National Corpus (BNC) with NLTK. They then ran two experiments on BERT and GPT-2, modeled on studies of human processing.

- **Experiment 1 (no context):** cosine similarity between each some-sentence and a semantic paraphrase ("Possibly all…") versus a pragmatic one ("Not all…"). Both models preferred the pragmatic reading, as humans do.
- **Experiment 2 (context):** the sentences were preceded by a Question Under Discussion (QUD), either upper-bound (*all*) or lower-bound (*any*), and scored with next sentence/token prediction. BERT behaved the same under both QUD types, which fits the Default model (Levinson, 2000). GPT-2 struggled with the QUD type that demands implicature, which matches the Context-driven model (Sperber and Wilson, 2002).

The work brings context-sensitive pragmatics into LLM evaluation.

## Key points
- Data: subject-position 'some + NP' sentences from the BNC, extracted with NLTK. The data and results are released publicly.
- Methods: cosine similarity (Experiment 1) and next sentence/token prediction (Experiment 2) on BERT and GPT-2.
- Without context, both models favour the *not all* implicature over the semantic reading.
- Under QUD context, BERT is insensitive to the question type (Default model). GPT-2 has difficulty when an upper-bound QUD demands the implicature (Context-driven model).
- Limitations noted by the authors:
  - Only one construction is tested: single-clause sentences with *some* in subject position.
  - The dataset is modest in size.
  - Only two older models are used.
  - The metrics are non-standard compared with NLI setups.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/)

Topics: [Implicature](https://md.research.elamur.ai/free/en/topics/implicature/)

## Cited by

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 11 not yet.

- [Are BabyLMs Deaf to Gricean Maxims? A Pragmatic Evaluation of Sample-efficient Language Models](https://md.research.elamur.ai/free/en/papers/askari-2025-are-babylms-deaf-gricean-maxims/)
- [Whose Pragmatics? Cultural Grounding as a Bottleneck for Stereotype Detection in Egyptian Arabic Social Media](https://md.research.elamur.ai/free/en/papers/assem-2026-whose-pragmatics-cultural-grounding-as/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{cho2024pragmatic,
  title = {{Pragmatic inference of scalar implicature by LLMs}},
  author = {Cho, Ye-eun and Kim, Seong mook},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-srw.2},
  eprint = {2408.06673},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.acl-srw.2}
}
```

- [← previous Pragmatic Competence Evaluation of Large Language Models for the Korean Language](https://md.research.elamur.ai/free/en/papers/park-2024-pragmatic-competence-evaluation-large-language/)
- [next → Prayer and (Im)Politeness: Reading Psalm 7](https://md.research.elamur.ai/free/en/papers/hbo-2025-jones-prayer-im-politeness-reading-psalm/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/3/)

## References

34 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ludivine E. Dupuy, Jean-Baptiste Van der Henst, Anne Cheylus and Anne C. Reboul. 2016. Context in generalized conversational implicatures: the case of some. Frontiers in Psychology, 7:381. [↗](https://doi.org/10.3389/fpsyg.2016.00381)
2. Douglas Bates, Martin Mächler, Benjamin M. Bolker and Steven C. Walker. 2014. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software, 67:1-48. [↗](https://doi.org/10.18637/jss.v067.i01) [our card](https://md.research.elamur.ai/free/en/papers/bates-2015-fitting-linear-mixed-effects-models/)
3. Bart Geurts. 2010. Quantity implicatures. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Bart+Geurts%2E+2010%2E+Quantity+implicatures%2E+Cambridge+University+Press%2E)
4. David Beaver and Brady Clark. 2008. Sense and sensitivity: How focus determines meaning. John Wiley & Sons. [find](https://scholar.google.com/scholar?q=David+Beaver+and+Brady+Clark%2E+2008%2E+Sense+and+sensitivity%3A+How+focus+determines+meaning%2E+John+Wiley+%26+Sons%2E)
5. Bart Geurts, Napoleon Katsos, Chris Cummins, Jonas Moons and Leo Noordman. 2010. Scalar quantifiers: Logic, acquisition, and processing. Language and cognitive processes, 25(1):130-148. [↗](https://doi.org/10.1080/01690960902955010)
6. Steven Bird. 2006. NLTK: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions. pages 69-72. 18 9 [find](https://scholar.google.com/scholar?q=Steven+Bird%2E+2006%2E+NLTK%3A+the+natural+language+toolkit%2E+In+Proceedings+of+the+COLING%2FACL+2006+Interactive+Presentation+Sessions%2E+pages+69%2D72%2E+18+9)
7. Bart Geurts and Rick Nouwen. 2007. 'At least'et al.: the semantics of scalar modifiers. Language, 83(3):533-559. [find](https://scholar.google.com/scholar?q=Bart+Geurts+and+Rick+Nouwen%2E+2007%2E+%27At+least%27et+al%2E%3A+the+semantics+of+scalar+modifiers%2E+Language%2C+83%283%29%3A533%2D559%2E)
8. Stephen Politzer-Ahles and Robert Fiorentino. 2013. The realization of scalar inferences: Context sensitivity without processing cost. PloS ONE, 8(5):e63943. [↗](https://doi.org/10.1371/journal.pone.0063943)
9. Herbert P Grice. 1975. Logic and conversation. In Speech acts (pp. 41-58). Brill. [↗](https://doi.org/10.1163/9789004368811_003)
10. Stephen Politzer-Ahles and E. Matthew Husband. 2018. Eye movement evidence for context-sensitive derivation of scalar inferences. Collabra: Psychology, 4(1):3. [↗](https://doi.org/10.1525/collabra.100)
11. John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second meeting of the north american chapter of the association for computational linguistics. (Volume 2). pages 159-166. [find](https://scholar.google.com/scholar?q=John+Hale%2E+2001%2E+A+probabilistic+Earley+parser+as+a+psycholinguistic+model%2E+In+Second+meeting+of+the+north+american+chapter+of+the+association+for+computational+linguistics%2E+%28Volume+2%29%2E+pages+159%2D166%2E) [our card](https://md.research.elamur.ai/free/en/papers/hale-2001-probabilistic-earley-parser-as-psycholinguistic/)
12. Jeffrey Pennington, Richard Socher and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 1532-1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher+and+Christopher+D+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Proceedings+of+the+2014+conference+on+empirical+methods+in+natural+language+proces)
13. Laurence Robert Horn. 1972. On the semantic properties of logical operators in English. University of California, Los Angeles. [find](https://scholar.google.com/scholar?q=Laurence+Robert+Horn%2E+1972%2E+On+the+semantic+properties+of+logical+operators+in+English%2E+University+of+California%2C+Los+Angeles%2E)
14. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
15. Jennifer Hu, Roger Levy, Judith Degen and Sebastian Schuster. 2023. Expectations over unspoken alternatives predict pragmatic inferences. Transactions of the Association for Computational Linguistics, 11, 885-901. [↗](https://doi.org/10.1162/tacl_a_00579) [our card](https://md.research.elamur.ai/free/en/papers/hu-2023-expectations-over-unspoken-alternatives-predict/)
16. Craige Roberts. 2004. Discourse context in dynamic interpretation. In L. Horn & G. Ward (Eds.), Handbook of contemporary pragmatic theory (pp. 197–220). Oxford: Blackwell. [find](https://scholar.google.com/scholar?q=Craige+Roberts%2E+2004%2E+Discourse+context+in+dynamic+interpretation%2E+In+L%2E+Horn+%26+G%2E+Ward+%28Eds%2E%29%2C+Handbook+of+contemporary+pragmatic+theory+%28pp%2E+197%E2%80%93220%29%2E+Oxford%3A+Blackwell%2E)
17. Yi Ting Huang and Jesse Snedeker. 2009. Online interpretation of scalar quantifiers: Insight into the semantics–pragmatics interface. Cognitive psychology, 58(3):376-415. [↗](https://doi.org/10.1016/j.cogpsych.2008.09.001)
18. Craige Roberts. 2012. Information structure: Towards an integrated formal theory of pragmatics. Semantics and Pragmatics, 5(6):1-69. [find](https://scholar.google.com/scholar?q=Craige+Roberts%2E+2012%2E+Information+structure%3A+Towards+an+integrated+formal+theory+of+pragmatics%2E+Semantics+and+Pragmatics%2C+5%286%29%3A1%2D69%2E)
19. Lamar Hunt III, Stephen Politzer-Ahles, Linzi Gibson, Utako Minai and Robert Fiorentino. 2013. Pragmatic inferences modulate N400 during sentence comprehension: Evidence from picture– sentence verification. Neuroscience Letters, 534:246-251. [↗](https://doi.org/10.1016/j.neulet.2012.11.044)
20. Eszter Ronai and Ming Xiang. 2021. Pragmatic inferences are QUD-sensitive: an experimental study. Journal of Linguistics, 57(4):841-870. [↗](https://doi.org/10.1017/s0022226720000389)
21. Sebastian Schuster, Yuxing Chen and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 5387-5403. [find](https://scholar.google.com/scholar?q=Sebastian+Schuster%2C+Yuxing+Chen+and+Judith+Degen%2E+2020%2E+Harnessing+the+linguistic+signal+to+predict+scalar+inferences%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Association+for+Computational+Li) [our card](https://md.research.elamur.ai/free/en/papers/schuster-2020-harnessing-linguistic-signal-predict-scalar/)
22. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan and Adina Williams. 2020. Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pages 8690-8705. [find](https://scholar.google.com/scholar?q=Paloma+Jeretic%2C+Alex+Warstadt%2C+Suvrat+Bhooshan+and+Adina+Williams%2E+2020%2E+Are+Natural+Language+Inference+Models+IMPPRESsive%3F+Learning+IMPlicature+and+PRESupposition%2E+In+Proceedings+of+the+58th+Annual+M) [our card](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/)
23. Dan Sperber and Deirdre Wilson. 2002. Pragmatics, modularity and mind-reading. Mind & Language, 17:3-23. [↗](https://doi.org/10.1111/1468-0017.00186)
24. Stephen C Levinson. 2000. Presumptive meanings: The theory of generalized conversational implicature. MIT press. [↗](https://doi.org/10.7551/mitpress/5526.001.0001)
25. John M. Tomlinson Jr, Todd M. Bailey and Lewis Bott. 2013. Possibly all of that and then some: Scalar implicatures are understood in two steps. Journal of memory and language, 69(1):18-35. [↗](https://doi.org/10.1016/j.jml.2013.02.003)
26. Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126-1177. [↗](https://doi.org/10.1016/j.cognition.2007.05.006)
27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+%C5%81ukasz+Kaiser+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+Advances+in+neural+information+processing+)
28. Elissa Li, Sebastian Schuster and Judith Degen. 2021. Predicting scalar inferences from “or” to “not both” using neural sentence encoders. In Proceedings of the Society for Computation in Linguistics 2021. pages 446-450. [↗](https://doi.org/10.7275/xr01-a852) [our card](https://md.research.elamur.ai/free/en/papers/li-2021-predicting-scalar-inferences-or-not/)
29. Deirdre Wilson and Dan Sperber. 1995. Relevance theory. In L. Horn & G. Ward (Eds.) The handbook of pragmatics (pp. 606-632). Oxford: Blackwell. [find](https://scholar.google.com/scholar?q=Deirdre+Wilson+and+Dan+Sperber%2E+1995%2E+Relevance+theory%2E+In+L%2E+Horn+%26+G%2E+Ward+%28Eds%2E%29+The+handbook+of+pragmatics+%28pp%2E+606%2D632%29%2E+Oxford%3A+Blackwell%2E)
30. Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv:1301.3781. [↗](https://arxiv.org/abs/1301.3781)
31. Ira Xiao Yang, Utako Minai and Robert Fiorentino. 2018. Context-sensitivity and individual differences in the derivation of scalar implicature. Frontiers in psychology, 9:1720. [↗](https://doi.org/10.3389/fpsyg.2018.01720)
32. A. Noveck and Andres Posada. 2003. Characterizing the time course of an implicature: An evoked potentials study. Brain and language, 85(2):203-210. 19 10 [↗](https://doi.org/10.1016/s0093-934x(03)00053-1)
33. Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card and Dan Jurafsky. 2022. Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2). pages 401-423. [find](https://scholar.google.com/scholar?q=Kaitlyn+Zhou%2C+Kawin+Ethayarajh%2C+Dallas+Card+and+Dan+Jurafsky%2E+2022%2E+Problems+with+Cosine+as+a+Measure+of+Embedding+Similarity+for+High+Frequency+Words%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the)
34. Arjen Zondervan, Luisa Meroni and Andrea Gualmini. 2008. Experiments on the role of the question under discussion for ambiguity resolution and implicature computation in adults. In Semantics and linguistic theory (Volume 18). pages 765-777. [↗](https://doi.org/10.3765/salt.v18i0.2486)