---
title: "We're Afraid Language Models Aren't Modeling Ambiguity (2023)"
description: "They release AmbiEnt, an English benchmark of 1,645 NLI pairs. 51 references, cited by 68 papers here."
authors: ["Alisa Liu", "Zhaofeng Wu", "Julian Michael", "Alane Suhr", "Peter West", "Alexander Koller", "Swabha Swayamdipta", "Noah Smith", "Yejin Choi"]
year: 2023
venue: "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2023.emnlp-main.51/"
doi: "10.18653/v1/2023.emnlp-main.51"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/liu-2023-we-re-afraid-language-models/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/liu-2023-we-re-afraid-language-models/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/) / We're Afraid Language Models Aren't Modeling Ambiguity

# We're Afraid Language Models Aren't Modeling Ambiguity

**Authors:** [Alisa Liu](https://md.research.elamur.ai/free/en/authors/alisa-liu/), [Zhaofeng Wu](https://md.research.elamur.ai/free/en/authors/zhaofeng-wu/), [Julian Michael](https://md.research.elamur.ai/free/en/authors/julian-michael/), [Alane Suhr](https://md.research.elamur.ai/free/en/authors/alane-suhr/), [Peter West](https://md.research.elamur.ai/free/en/authors/peter-west/), [Alexander Koller](https://md.research.elamur.ai/free/en/authors/alexander-koller/), [Swabha Swayamdipta](https://md.research.elamur.ai/free/en/authors/swabha-swayamdipta/), [Noah Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) and [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) · **Year:** 2023 · **Venue:** [Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2023](https://md.research.elamur.ai/free/en/venues/emnlp/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.emnlp-main.51/) · [PDF](https://aclanthology.org/2023.emnlp-main.51.pdf) · [DOI 10.18653/v1/2023.emnlp-main.51](https://doi.org/10.18653/v1/2023.emnlp-main.51) · [OpenAlex](https://openalex.org/W4389524313)

**Citations:** 40 citations (OpenAlex, 2026-10-14) · [68](https://md.research.elamur.ai/free/en/papers/liu-2023-we-re-afraid-language-models/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper asks whether pretrained language models can notice when a sentence has more than one plausible meaning and separate those meanings. Formally listing every interpretation is often impractical, so the authors take a functional view. They treat ambiguity as anything in a premise or hypothesis that changes the natural language inference (NLI) label. They release AmbiEnt, an English benchmark of 1,645 NLI pairs. Each pair has a set of gold labels, plus a disambiguating rewrite for every reading when several apply. The data comes from two sources: a small hand-curated set based on textbook phenomena, and a larger pool of InstructGPT-generated pairs annotated by linguists. Tests built on AmbiEnt probe whether LMs can produce disambiguations, recognize valid readings, and reflect alternative interpretations in their continuations. Even GPT-4 performs poorly on these tests. Multilabel NLI classifiers also struggle to predict the full label set. A case study shows that such a classifier can still help spot political claims that mislead through ambiguity. The work argues that benchmarks assuming one meaning per sentence hide an important gap in model understanding.

## Key points
- **Dataset composition:** AmbiEnt has 1,645 examples, and 35.2% of them carry multiple labels. Pragmatic ambiguity is the most frequent type (about 44.8%), followed by lexical ambiguity (about 20.0%). Syntactic, scopal, coreference, figurative and other types are also included.
- **Collection pipeline:** The authors hand-curated 142 examples from sources such as DistNLI, IMPPRES, NLI Diagnostics, MNLI and WANLI. The remaining pairs were generated by InstructGPT from prompts built on WANLI neighbor groups. These were kept only if a multilabel RoBERTa-large model gave probability ≥0.05 to more than one label. Two linguists then annotated each pair independently, and an author merged their annotations.
- **Crowdworker check:** Nine crowdworkers labeled each example under the usual single-label scheme. Their disagreement on ambiguous items largely disappeared once they saw the disambiguated versions.
- **GPT-4 results:** In crowdworker evaluation, only 32% of GPT-4's disambiguations were judged correct. Disambiguations written for the dataset were judged correct 90% of the time. GPT-4 led two of the three tests, while the smallest FLAN-T5 led the third.
- **Multilabel NLI results:** The best finetuned multilabel NLI model recovered the exact label set in just 43.6% of cases. Even so, it flagged political claims that fact-checkers had marked as ambiguous, and it surfaced new ambiguous claims as well. Data and code are available at github.com/alisawuffles/ambient.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations
- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 127 works · 212,523 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 219 works · 89,712 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations
- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 159 works · 65,157 citations
- [University of Southern California](https://md.research.elamur.ai/free/en/universities/university-of-southern-california/) — University, Los Angeles, United States · 104 works · 33,876 citations
- [Saarland University](https://md.research.elamur.ai/free/en/universities/saarland-university/) — University, Saarbrücken, Germany · 102 works · 17,447 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith and Yejin Choi. 2023. We're Afraid Language Models Aren't Modeling Ambiguity. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{liu2023we,
  title = {{We're Afraid Language Models Aren't Modeling Ambiguity}},
  author = {Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah Smith and Yejin Choi},
  booktitle = {Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.51},
  url = {https://doi.org/10.18653/v1/2023.emnlp-main.51}
}
```

- [← previous RestGPT: Connecting Large Language Models with Real-World RESTful APIs](https://md.research.elamur.ai/free/en/papers/song-2023-restgpt-connecting-large-language-models/)
- [next → Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest](https://md.research.elamur.ai/free/en/papers/hessel-2023-do-androids-laugh-electric-sheep/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/6/)

## References

51 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Pangbo Ban, Yifan Jiang, Tianran Liu, and Shane Steinert-Threlkeld. 2022. Testing pre-trained language models’ understanding of distributivity via causal mediation analysis. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 314–324, Abu Dha… [find](https://scholar.google.com/scholar?q=Pangbo+Ban%2C+Yifan+Jiang%2C+Tianran+Liu%2C+and+Shane+Steinert%2DThrelkeld%2E+2022%2E+Testing+pre%2Dtrained+language+models%E2%80%99+understanding+of+distributivity+via+causal+mediation+analysis%2E+In+Proceedings+of+the+Fift)
2. Beata Beigman Klebanov and Eyal Beigman. 2009. From annotator agreement to noise models. Computational Linguistics, 35(4):495–503. [find](https://scholar.google.com/scholar?q=Beata+Beigman+Klebanov+and+Eyal+Beigman%2E+2009%2E+From+annotator+agreement+to+noise+models%2E+Computational+Linguistics%2C+35%284%29%3A495%E2%80%93503%2E)
3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati… [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
4. Andrew Carnie. 2013. Syntax: A Generative Introduction. Introducing Linguistics. Wiley. [find](https://scholar.google.com/scholar?q=Andrew+Carnie%2E+2013%2E+Syntax%3A+A+Generative+Introduction%2E+Introducing+Linguistics%2E+Wiley%2E)
5. Jifan Chen, Aniruddh Sriram, Eunsol Choi, and Greg Durrett. 2022. Generating literal and implied sub-questions to fact-check complex claims. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3495–3516, Abu Dhabi, United Arab Emirates. Association for Co… [find](https://scholar.google.com/scholar?q=Jifan+Chen%2C+Aniruddh+Sriram%2C+Eunsol+Choi%2C+and+Greg+Durrett%2E+2022%2E+Generating+literal+and+implied+sub%2Dquestions+to+fact%2Dcheck+complex+claims%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+)
6. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. [Uncertain natural language inference](https://md.research.elamur.ai/free/en/papers/chen-2020-uncertain-natural-language-inference/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online. Association for Computational Linguistics.
7. [Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan…](https://md.research.elamur.ai/free/en/papers/chung-2022-scaling-instruction-finetuned-language-models/)
8. Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Linguistics%2C+pages+8690%E2%80%938705%2C+Online%2E+Association+for+Computational+Linguistics%2E)
9. Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. [Investigating Reasons for Disagreement in Natural Language Inference](https://md.research.elamur.ai/free/en/papers/jiang-2022-investigating-reasons-disagreement-natural-language/). Transactions of the Association for Computational Linguistics, 10:1357– 1374.
10. Kate Kearns. 2000. Semantics. St. Martin’s Press. [find](https://scholar.google.com/scholar?q=Kate+Kearns%2E+2000%2E+Semantics%2E+St%2E+Martin%E2%80%99s+Press%2E)
11. Alexander Koller, Michaela Regneri, and Stefan Thater. 2008. Regular tree grammars as a formalism for scope underspecification. In Proceedings of ACL-08: HLT, pages 218–226, Columbus, Ohio. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexander+Koller%2C+Michaela+Regneri%2C+and+Stefan+Thater%2E+2008%2E+Regular+tree+grammars+as+a+formalism+for+scope+underspecification%2E+In+Proceedings+of+ACL%2D08%3A+HLT%2C+pages+218%E2%80%93226%2C+Columbus%2C+Ohio%2E+Associatio)
12. Dmitrii Krasheninnikov, Egor Krasheninnikov, and David Krueger. 2022. Assistance with large language models. In Proceedings of the ML Safety Workshop at NeurIPS. [find](https://scholar.google.com/scholar?q=Dmitrii+Krasheninnikov%2C+Egor+Krasheninnikov%2C+and+David+Krueger%2E+2022%2E+Assistance+with+large+language+models%2E+In+Proceedings+of+the+ML+Safety+Workshop+at+NeurIPS%2E)
13. Kenneth Church and Ramesh Patil. 1982. Coping with syntactic ambiguity or how to put the block in the box on the table. American Journal of Computational Linguistics, 8(3-4):139–149. [find](https://scholar.google.com/scholar?q=Kenneth+Church+and+Ramesh+Patil%2E+1982%2E+Coping+with+syntactic+ambiguity+or+how+to+put+the+block+in+the+box+on+the+table%2E+American+Journal+of+Computational+Linguistics%2C+8%283%2D4%29%3A139%E2%80%93149%2E)
14. Jeremy R. Cole, Michael J. Q. Zhang, Daniel Gillick, Julian Martin Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. 2023. [Selectively answering ambiguous questions](https://md.research.elamur.ai/free/en/papers/cole-2023-selectively-answering-ambiguous-questions/).
15. Ann Copestake and Dan Flickinger. 2000. An open source grammar development environment and broad-coverage English grammar using HPSG. In Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Ann+Copestake+and+Dan+Flickinger%2E+2000%2E+An+open+source+grammar+development+environment+and+broad%2Dcoverage+English+grammar+using+HPSG%2E+In+Proceedings+of+the+Second+International+Conference+on+Language+)
16. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. [RealToxicityPrompts: Evaluating neural toxic degeneration in language models](https://md.research.elamur.ai/free/en/papers/gehman-2020-realtoxicityprompts-evaluating-neural-toxic-degeneration/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguisti…
17. Mitchell L. Gordon, Michelle S. Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S. Bernstein. 2022. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, Ne… [find](https://scholar.google.com/scholar?q=Mitchell+L%2E+Gordon%2C+Michelle+S%2E+Lam%2C+Joon+Sung+Park%2C+Kayur+Patel%2C+Jeff+Hancock%2C+Tatsunori+Hashimoto%2C+and+Michael+S%2E+Bernstein%2E+2022%2E+Jury+learning%3A+Integrating+dissenting+voices+into+machine+learning+)
18. Barbara J. Grosz. 1977. The Representation and Use of Focus in Dialogue Understanding. Ph.D. thesis. [find](https://scholar.google.com/scholar?q=Barbara+J%2E+Grosz%2E+1977%2E+The+Representation+and+Use+of+Focus+in+Dialogue+Understanding%2E+Ph%2ED%2E+thesis%2E)
19. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ari+Holtzman%2C+Jan+Buys%2C+Li+Du%2C+Maxwell+Forbes%2C+and+Yejin+Choi%2E+2020%2E+The+curious+case+of+neural+text+degeneration%2E+In+International+Conference+on+Learning+Representations%2E)
20. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. [Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition](https://md.research.elamur.ai/free/en/papers/jeretic-2020-are-natural-language-inference-models/). In Proceedings of the 58th Annual Meeting of the Association for Computational Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. C…
21. Mina Lee, Percy Liang, and Qian Yang. 2022. Coauthor: Designing a human-AI collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA. [find](https://scholar.google.com/scholar?q=Mina+Lee%2C+Percy+Liang%2C+and+Qian+Yang%2E+2022%2E+Coauthor%3A+Designing+a+human%2DAI+collaborative+writing+dataset+for+exploring+language+model+capabilities%2E+In+CHI+Conference+on+Human+Factors+in+Computing+Syst)
22. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. [WANLI: Worker and AI collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computat…
23. Johannes Mario Meissner, Napat Thumwanit, Saku Sugawara, and Akiko Aizawa. 2021. [Embracing ambiguity: Shifting the training target of NLI models](https://md.research.elamur.ai/free/en/papers/meissner-2021-embracing-ambiguity-shifting-training-target/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language…
24. Meta. 2022. No language left behind: Scaling human-centered machine translation. [find](https://scholar.google.com/scholar?q=Meta%2E+2022%2E+No+language+left+behind%3A+Scaling+human%2Dcentered+machine+translation%2E)
25. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+2020%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+La)
26. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. [What can we learn from collective human opinions on natural language inference data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics.
27. OpenAI. 2022. Introducing ChatGPT. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Introducing+ChatGPT%2E)
28. OpenAI. 2023. [GPT-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
29. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
30. Ellie Pavlick and Tom Kwiatkowski. 2019. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, 7:677–694.
31. Sandro Pezzelle. 2023. Dealing with semantic underspecification in multimodal NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12098–12112, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sandro+Pezzelle%2E+2023%2E+Dealing+with+semantic+underspecification+in+multimodal+NLP%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers%29%2C+p)
32. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3):280–291. [find](https://scholar.google.com/scholar?q=Steven+T%2E+Piantadosi%2C+Harry+Tily%2C+and+Edward+Gibson%2E+2012%2E+The+communicative+function+of+ambiguity+in+language%2E+Cognition%2C+122%283%29%3A280%E2%80%93291%2E)
33. Barbara Plank. 2022. [The “problem” of human label variation: On ground truth in data, modeling and evaluation](https://md.research.elamur.ai/free/en/papers/plank-2022-problem-human-label-variation-ground/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
34. Massimo Poesio and Ron Artstein. 2005. [The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account](https://md.research.elamur.ai/free/en/papers/poesio-2005-reliability-anaphoric-annotation-reconsidered-taking/). In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83, Ann Arbor, Michigan. Association for Computational Linguistics.
35. Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022. [Annotators with attitudes: How annotator beliefs and identities bias toxic language detection](https://md.research.elamur.ai/free/en/papers/sap-2022-annotators-attitudes-how-annotator-beliefs/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling…
36. Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019. [The woman worked as a babysitter: On biases in language generation](https://md.research.elamur.ai/free/en/papers/sheng-2019-woman-worked-as-babysitter-biases/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Process…
37. Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. 2022. [Blenderbot 3: a deployed conversational…](https://md.research.elamur.ai/free/en/papers/shuster-2022-blenderbot-3-deployed-conversational-agent/)
38. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, page 4444–4451. AAAI Press. [find](https://scholar.google.com/scholar?q=Robyn+Speer%2C+Joshua+Chin%2C+and+Catherine+Havasi%2E+2017%2E+Conceptnet+5%2E5%3A+An+open+multilingual+graph+of+general+knowledge%2E+In+Proceedings+of+the+Thirty%2DFirst+AAAI+Conference+on+Artificial+Intelligence%2C+pa)
39. Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme. 2023. Why did the chicken cross the road? Rephrasing and analyzing ambiguous questions in VQA. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Elias+Stengel%2DEskin%2C+Jimena+Guallar%2DBlasco%2C+Yi+Zhou%2C+and+Benjamin+Van+Durme%2E+2023%2E+Why+did+the+chicken+cross+the+road%3F+Rephrasing+and+analyzing+ambiguous+questions+in+VQA%2E+In+Proceedings+of+the+61th+A)
40. Alex Tamkin, Kunal Handa, Avash Shrestha, and Noah Goodman. 2023. Task ambiguity in humans and language models. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Alex+Tamkin%2C+Kunal+Handa%2C+Avash+Shrestha%2C+and+Noah+Goodman%2E+2023%2E+Task+ambiguity+in+humans+and+language+models%2E+In+The+Eleventh+International+Conference+on+Learning+Representations%2E)
41. Lewis Thomas. 1974. The lives of a cell. Notes of a biology watcher, New york (The Viking Press) 1974. [find](https://scholar.google.com/scholar?q=Lewis+Thomas%2E+1974%2E+The+lives+of+a+cell%2E+Notes+of+a+biology+watcher%2C+New+york+%28The+Viking+Press%29+1974%2E)
42. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. [LLaMA: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/).
43. Alexandra Uma, Tommaso Fornaciari, Anca Dumitrache, Tristan Miller, Jon Chamberlain, Barbara Plank, Edwin Simpson, and Massimo Poesio. 2021. SemEval-2021 task 12: Learning with disagreements. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 338– 347, Onl… [find](https://scholar.google.com/scholar?q=Alexandra+Uma%2C+Tommaso+Fornaciari%2C+Anca+Dumitrache%2C+Tristan+Miller%2C+Jon+Chamberlain%2C+Barbara+Plank%2C+Edwin+Simpson%2C+and+Massimo+Poesio%2E+2021%2E+SemEval%2D2021+task+12%3A+Learning+with+disagreements%2E+In+Proce)
44. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. [GLUE: A multi-task benchmark and analysis platform for natural language understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,…
45. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
46. Yuewei Yuan, Chaitanya Malaviya, and Mark Yatskar. 2023. [AmbiCoref: Evaluating human and model sensitivity to ambiguous coreference](https://md.research.elamur.ai/free/en/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/). In Findings of the Association for Computational Linguistics: EACL 2023, pages 1023–1030, Dubrovnik, Croatia. Association for Computational Linguistics.
47. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. [Ordinal common-sense inference](https://md.research.elamur.ai/free/en/papers/zhang-2017-ordinal-common-sense-inference/). Transactions of the Association for Computational Linguistics, 5:379–395.
48. Shujian Zhang, Chengyue Gong, and Eunsol Choi. 2021. Learning with different amounts of annotation: From zero to many labels. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7620–7632, Online and Punta Cana, Dominican Republic. Association for Computa… [find](https://scholar.google.com/scholar?q=Shujian+Zhang%2C+Chengyue+Gong%2C+and+Eunsol+Choi%2E+2021%2E+Learning+with+different+amounts+of+annotation%3A+From+zero+to+many+labels%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Lang)
49. Xinliang Frederick Zhang and Marie-Catherine de Marneffe. 2021. [Identifying inherent disagreement in natural language inference](https://md.research.elamur.ai/free/en/papers/zhang-2021-identifying-inherent-disagreement-natural-language/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4908–4915, Online.…
50. Xiang Zhou, Yixin Nie, and Mohit Bansal. 2022. [Distributed NLI: Learning to predict human opinion distributions for language reasoning](https://md.research.elamur.ai/free/en/papers/zhou-2022-distributed-nli-learning-predict-human/). In Findings of the Association for Computational Linguistics: ACL 2022, pages 972–987, Dublin, Ireland. Association for Computational Linguistics.
51. George Kingsley Zipf. 1949. Human behavior and the principle of least effort. [find](https://scholar.google.com/scholar?q=George+Kingsley+Zipf%2E+1949%2E+Human+behavior+and+the+principle+of+least+effort%2E)

*Updated 24 October 2026*