---
title: "Zero and Few-shot Semantic Parsing with Ambiguous Inputs"
description: "They propose three new metrics and compare model outputs with crowdsourced human judgements. 64 references, cited by 13 papers here."
authors: ["Elias Stengel-Eskin", "Kyle Rawlins", "Benjamin Van Durme"]
year: 2023
venue: "arXiv:2306.00824"
source: "https://arxiv.org/abs/2306.00824"
doi: "10.48550/arxiv.2306.00824"
arxiv: "2306.00824"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/stengel-eskin-2023-zero-few-shot-semantic-parsing/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/stengel-eskin-2023-zero-few-shot-semantic-parsing/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / Zero and Few-shot Semantic Parsing with Ambiguous Inputs

# Zero and Few-shot Semantic Parsing with Ambiguous Inputs

**Authors:** [Elias Stengel-Eskin](https://md.research.elamur.ai/free/en/authors/elias-stengel-eskin/), [Kyle Rawlins](https://md.research.elamur.ai/free/en/authors/kyle-rawlins/) and [Benjamin Van Durme](https://md.research.elamur.ai/free/en/authors/benjamin-van-durme/) · **Year:** 2023 · **Venue:** arXiv:2306.00824 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2306.00824) · [PDF](https://arxiv.org/pdf/2306.00824) · [DOI 10.48550/arxiv.2306.00824](https://doi.org/10.48550/arxiv.2306.00824) · [arXiv 2306.00824](https://arxiv.org/abs/2306.00824) · [OpenAlex](https://openalex.org/W4379261285)

**Citations:** 2 citations (OpenAlex, 2026-10-14) · 13 in our library, of the 149,439 papers whose bibliographies we read

## Summary
Semantic parsing benchmarks usually assume each utterance has exactly one correct formal meaning, so it has been unclear whether parsers built on large language models represent ambiguity at all. Stengel-Eskin, Rawlins and Van Durme address this with AMP (Ambiguous Parsing), a template-based framework, dataset and challenge. In AMP, each ambiguous sentence is paired with two logical forms, written in first-order logic (FOL) or Lisp. The templates cover five classic ambiguity types and also generate unambiguous controls.

The authors use in-context learning to test several few-shot text-to-code models in two settings:
- **Zero-shot:** the prompt supplies the building blocks for both readings but no example of the ambiguity itself.
- **Mixed-prompt:** the prompt contains conflicting examples for each reading in varying proportions.

They propose three new metrics and compare model outputs with crowdsourced human judgements. Models mostly fail at zero-shot composition and seldom produce both parses. However, some closely follow the proportions of readings shown in conflicting prompts. The authors conclude that datasets should explicitly include ambiguity and that evaluation should consider distributions over outputs.

## Key points
- AMP covers five ambiguity types: prepositional-phrase (PP) attachment, quantifier scope, inverse scope, pronoun coreference (bound), and conjunctions. Logical forms are canonicalized by ordering binary trees and anonymizing variables, so logically equivalent forms count as identical.
- Locally run models use grammar-constrained decoding via BenchCLAMP with a beam size of 5. Every output is therefore well-formed, which separates semantic errors from syntactic ones.
- An error analysis of Codegen-16B sampled 10 wrong outputs per ambiguity type:
  - Inverse scope: all 10 put ∃ where ∀ was needed.
  - PP attachment: 9 of 10 omitted a predicate.
  - Bound: 8 of 10 reused one event variable for two verbs.
- Human judgements came from Mechanical Turk annotators paid about $14/hour. Each did 20 items and rated confidence on a slider, which was mapped to probabilities between 0.5 and 1.0. Visual verb–noun pairs (e.g., saw–telescope) favoured the instrument reading, while tactile ones (e.g., held–gloves) favoured possession.
- According to the authors, no earlier work had examined how in-context learning behaves when prompts contain contradictory evidence. Data and code are released at github.com/esteng/ambiguous_parsing.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Johns Hopkins University](https://md.research.elamur.ai/free/en/universities/johns-hopkins-university/) — University, Baltimore, United States · 240 works · 96,487 citations
- [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 67 works · 16,474 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (1): [🧾 Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) · 185 works

## Cite

Cite as: Elias Stengel-Eskin, Kyle Rawlins and Benjamin Van Durme. 2023. Zero and Few-shot Semantic Parsing with Ambiguous Inputs. arXiv:2306.00824.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{stengeleskin2023zero,
  title = {{Zero and Few-shot Semantic Parsing with Ambiguous Inputs}},
  author = {Elias Stengel-Eskin, Kyle Rawlins and Benjamin {Van Durme}},
  year = {2023},
  doi = {10.48550/arxiv.2306.00824},
  eprint = {2306.00824},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.00824}
}
```

- [← previous WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models](https://md.research.elamur.ai/free/en/papers/benchekroun-2023-worldsense-synthetic-benchmark-grounded-reasoning/)
- [next → AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference](https://md.research.elamur.ai/free/en/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/)

## References

64 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Scott AnderBois, Adrian Brasoveanu, and Robert Henderson. The pragmatics of quantifier scope: A corpus study. In Proceedings of Sinn und Bedeutung, volume 16, pp. 15–28, 2012. [find](https://scholar.google.com/scholar?q=Scott+AnderBois%2C+Adrian+Brasoveanu%2C+and+Robert+Henderson%2E+The+pragmatics+of+quantifier+scope%3A+A+corpus+study%2E+In+Proceedings+of+Sinn+und+Bedeutung%2C+volume+16%2C+pp%2E+15%E2%80%9328%2C+2012%2E)
2. Yoav Artzi and Luke Zettlemoyer. [Weakly supervised learning of semantic parsers for mapping instructions to actions](https://md.research.elamur.ai/free/en/papers/artzi-2013-weakly-supervised-learning-semantic-parsers/). Transactions of the Association for Computational Linguistics, 1:49–62, 2013.
3. Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. [Broad-coverage ccg semantic parsing with amr](https://md.research.elamur.ai/free/en/papers/artzi-2015-broad-coverage-ccg-semantic-parsing/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1699–1710, 2015.
4. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1533–1544, 2013.
5. Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, and Shimon Ullman. Do you see what I mean? Visual resolution of linguistic ambiguities. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1477–1487, 2015. [find](https://scholar.google.com/scholar?q=Yevgeni+Berzak%2C+Andrei+Barbu%2C+Daniel+Harari%2C+Boris+Katz%2C+and+Shimon+Ullman%2E+Do+you+see+what+I+mean%3F+Visual+resolution+of+linguistic+ambiguities%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Meth)
6. Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different answers? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4271–4280, 2019. [find](https://scholar.google.com/scholar?q=Nilavra+Bhattacharya%2C+Qing+Li%2C+and+Danna+Gurari%2E+Why+does+a+visual+question+have+different+answers%3F+In+Proceedings+of+the+IEEE%2FCVF+International+Conference+on+Computer+Vision%2C+pp%2E+4271%E2%80%934280%2C+2019%2E)
7. Johan Bos. Computational semantics in discourse: Underspecification, resolution, and inference. Journal of Logic, Language and Information, 13:139–157, 2004. [find](https://scholar.google.com/scholar?q=Johan+Bos%2E+Computational+semantics+in+discourse%3A+Underspecification%2C+resolution%2C+and+inference%2E+Journal+of+Logic%2C+Language+and+Information%2C+13%3A139%E2%80%93157%2C+2004%2E)
8. Glenn W Brier et al. Verification of forecasts expressed in terms of probability. Monthly weather review, 78(1):1–3, 1950. [find](https://scholar.google.com/scholar?q=Glenn+W+Brier+et+al%2E+Verification+of+forecasts+expressed+in+terms+of+probability%2E+Monthly+weather+review%2C+78%281%29%3A1%E2%80%933%2C+1950%2E)
9. Alfonso Caramazza, Ellen Grober, Catherine Garvey, and Jack Yates. Comprehension of anaphoric pronouns. Journal of verbal learning and verbal behavior, 16(5):601–609, 1977. [find](https://scholar.google.com/scholar?q=Alfonso+Caramazza%2C+Ellen+Grober%2C+Catherine+Garvey%2C+and+Jack+Yates%2E+Comprehension+of+anaphoric+pronouns%2E+Journal+of+verbal+learning+and+verbal+behavior%2C+16%285%29%3A601%E2%80%93609%2C+1977%2E)
10. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. [Uncertain natural language inference](https://md.research.elamur.ai/free/en/papers/chen-2020-uncertain-natural-language-inference/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8772–8779, Online, July 2020. Association for Computational Linguistics. doi: 1…
11. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL [https://lmsys.org/blog/2023-03-30-vicuna…](https://lmsys.org/blog/2023-03-30-vicuna…) [↗](https://lmsys.org/blog/2023-03-30-vicuna…)
12. Ann Copestake, Dan Flickinger, Carl Pollard, and Ivan A Sag. Minimal recursion semantics: An introduction. Research on language and computation, 3:281–332, 2005. [find](https://scholar.google.com/scholar?q=Ann+Copestake%2C+Dan+Flickinger%2C+Carl+Pollard%2C+and+Ivan+A+Sag%2E+Minimal+recursion+semantics%3A+An+introduction%2E+Research+on+language+and+computation%2C+3%3A281%E2%80%93332%2C+2005%2E)
13. Marco Damonte, Rahul Goel, and Tagyoung Chung. Practical semantic parsing for spoken language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pp. 16–23, 2019. [find](https://scholar.google.com/scholar?q=Marco+Damonte%2C+Rahul+Goel%2C+and+Tagyoung+Chung%2E+Practical+semantic+parsing+for+spoken+language+understanding%2E+In+Proceedings+of+the+2019+Conference+of+the+North+American+Chapter+of+the+Association+for+)
14. Li Dong and Mirella Lapata. [Language to logical form with neural attention](https://md.research.elamur.ai/free/en/papers/dong-2016-language-logical-form-neural-attention/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 33–43, Berlin, Germany, 2016. Association for Computational Linguistics. doi: 10.18653/v1/P16-1004. URL…
15. Veena D Dwivedi. Interpreting quantifier scope ambiguity: Evidence of heuristic first, algorithmic second processing. PloS one, 8(11):e81461, 2013. [find](https://scholar.google.com/scholar?q=Veena+D+Dwivedi%2E+Interpreting+quantifier+scope+ambiguity%3A+Evidence+of+heuristic+first%2C+algorithmic+second+processing%2E+PloS+one%2C+8%2811%29%3Ae81461%2C+2013%2E)
16. Ruth Filik, Kevin B Paterson, and Simon P Liversedge. Processing doubly quantified sentences: Evidence from eye movements. Psychonomic Bulletin & Review, 11(5):953–959, 2004. [find](https://scholar.google.com/scholar?q=Ruth+Filik%2C+Kevin+B+Paterson%2C+and+Simon+P+Liversedge%2E+Processing+doubly+quantified+sentences%3A+Evidence+from+eye+movements%2E+Psychonomic+Bulletin+%26+Review%2C+11%285%29%3A953%E2%80%93959%2C+2004%2E)
17. Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71, 1988. [find](https://scholar.google.com/scholar?q=Jerry+A+Fodor+and+Zenon+W+Pylyshyn%2E+Connectionism+and+cognitive+architecture%3A+A+critical+analysis%2E+Cognition%2C+28%281%2D2%29%3A3%E2%80%9371%2C+1988%2E)
18. Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoı̂t Sagot, and Rachel Bawden. Tackling ambiguity with images: Improved multimodal machine translation and contrastive evaluation. arXiv preprint arXiv:2212.10140, 2022. [find](https://scholar.google.com/scholar?q=Matthieu+Futeral%2C+Cordelia+Schmid%2C+Ivan+Laptev%2C+Beno%C4%B1%CC%82t+Sagot%2C+and+Rachel+Bawden%2E+Tackling+ambiguity+with+images%3A+Improved+multimodal+machine+translation+and+contrastive+evaluation%2E+arXiv+preprint+arX)
19. Rohit J Kate, Yuk Wah Wong, Raymond J Mooney, et al. Learning to transform natural to formal languages. In AAAI, volume 5, pp. 1062–1068, 2005. [find](https://scholar.google.com/scholar?q=Rohit+J+Kate%2C+Yuk+Wah+Wong%2C+Raymond+J+Mooney%2C+et+al%2E+Learning+to+transform+natural+to+formal+languages%2E+In+AAAI%2C+volume+5%2C+pp%2E+1062%E2%80%931068%2C+2005%2E)
20. Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9087–9105, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/202… [find](https://scholar.google.com/scholar?q=Najoung+Kim+and+Tal+Linzen%2E+COGS%3A+A+compositional+generalization+challenge+based+on+semantic+interpretation%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28)
21. Howard S Kurtzman and Maryellen C MacDonald. Resolution of quantifier scope ambiguities. Cognition, 48(3):243–279, 1993. [find](https://scholar.google.com/scholar?q=Howard+S+Kurtzman+and+Maryellen+C+MacDonald%2E+Resolution+of+quantifier+scope+ambiguities%2E+Cognition%2C+48%283%29%3A243%E2%80%93279%2C+1993%2E)
22. James R Lackner and Merrill F Garrett. Resolving ambiguity: Effects of biasing context in the unattended ear. Cognition, 1(4):359–372, 1972. [find](https://scholar.google.com/scholar?q=James+R+Lackner+and+Merrill+F+Garrett%2E+Resolving+ambiguity%3A+Effects+of+biasing+context+in+the+unattended+ear%2E+Cognition%2C+1%284%29%3A359%E2%80%93372%2C+1972%2E)
23. Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stock-holmsmässan, Stockholm,… [find](https://scholar.google.com/scholar?q=Brenden+M%2E+Lake+and+Marco+Baroni%2E+Generalization+without+systematicity%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+In+Jennifer+G%2E+Dy+and+Andreas+Krause+%28eds%2E%29%2C+Proceedings)
24. PMLR, 2018. URL [http://proceedings.mlr.press/v80/](http://proceedings.mlr.press/v80/) lake18a.html . [↗](http://proceedings.mlr.press/v80/)
25. Alisa Liu, Zhaofeng Wu, Julian Michael, Alane Suhr, Peter West, Alexander Koller, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. We’re afraid language models aren’t modeling ambiguity. arXiv preprint arXiv:2304.[14399, 2023](https://md.research.elamur.ai/free/en/papers/liu-2023-we-re-afraid-language-models/).
26. Ninareh Mehrabi, Palash Goyal, Apurv Verma, Jwala Dhamala, Varun Kumar, Qian Hu, Kai-Wei Chang, Richard Zemel, Aram Galstyan, and Rahul Gupta. Is the elephant flying? resolving ambiguities in text-to-image generative models. arXiv preprint arXiv:2211.12503, 2022. [find](https://scholar.google.com/scholar?q=Ninareh+Mehrabi%2C+Palash+Goyal%2C+Apurv+Verma%2C+Jwala+Dhamala%2C+Varun+Kumar%2C+Qian+Hu%2C+Kai%2DWei+Chang%2C+Richard+Zemel%2C+Aram+Galstyan%2C+and+Rahul+Gupta%2E+Is+the+elephant+flying%3F+resolving+ambiguities+in+text%2Dto%2D)
27. Grégoire Mialon, Roberto Dessı̀, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. [Augmented language models: a survey](https://md.research.elamur.ai/free/en/papers/mialon-2023-augmented-language-models-survey/). arXiv preprint arXiv:2302.07842, 2023.
28. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5783–5797, 2020. [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Julian+Michael%2C+Hannaneh+Hajishirzi%2C+and+Luke+Zettlemoyer%2E+AmbigQA%3A+Answering+ambiguous+open%2Ddomain+questions%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language)
29. Richard Montague. English as a formal language. Linguaggi nella Societa e nella Tecnica, 1970. [find](https://scholar.google.com/scholar?q=Richard+Montague%2E+English+as+a+formal+language%2E+Linguaggi+nella+Societa+e+nella+Tecnica%2C+1970%2E)
30. Yixin Nie, Xiang Zhou, and Mohit Bansal. [What can we learn from collective human opinions on natural language inference data?](https://md.research.elamur.ai/free/en/papers/nie-2020-what-can-we-learn-collective/) In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2020.
31. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203.13474, 2022. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+Codegen%3A+An+open+large+language+model+for+code+with+multi%2Dturn+program+synthesis%2E+arXiv+pre)
32. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). Advances in Neural Information Processing Systems, 35: 27730–27744, 2022.
33. Aaron Parisi, Yao Zhao, and Noah Fiedel. [Talm: Tool augmented language models](https://md.research.elamur.ai/free/en/papers/parisi-2022-talm-tool-augmented-language-models/). arXiv preprint arXiv:2205.12255, 2022.
34. Terence Parsons. Events in the Semantics of English: A Study in Subatomic Semantics. MIT Press, 1990. [find](https://scholar.google.com/scholar?q=Terence+Parsons%2E+Events+in+the+Semantics+of+English%3A+A+Study+in+Subatomic+Semantics%2E+MIT+Press%2C+1990%2E)
35. Ellie Pavlick and Tom Kwiatkowski. [Inherent disagreements in human textual inferences](https://md.research.elamur.ai/free/en/papers/pavlick-2019-inherent-disagreements-human-textual-inferences/). Transactions of the Association for Computational Linguistics, 7:677–694, 2019.
36. Steven T Piantadosi, Harry Tily, and Edward Gibson. The communicative function of ambiguity in language. Cognition, 122(3):280–291, 2012. [find](https://scholar.google.com/scholar?q=Steven+T+Piantadosi%2C+Harry+Tily%2C+and+Edward+Gibson%2E+The+communicative+function+of+ambiguity+in+language%2E+Cognition%2C+122%283%29%3A280%E2%80%93291%2C+2012%2E)
37. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners, 2019. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+Language+models+are+unsupervised+multitask+learners%2C+2019%2E)
38. Nathan Ellis Rasmussen and William Schuler. A corpus of encyclopedia articles with logical forms. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020. [find](https://scholar.google.com/scholar?q=Nathan+Ellis+Rasmussen+and+William+Schuler%2E+A+corpus+of+encyclopedia+articles+with+logical+forms%2E+In+Proceedings+of+the+12th+Conference+on+Language+Resources+and+Evaluation+%28LREC+2020%29%2C+2020%2E)
39. Keith Rayner, Marcia Carlson, and Lyn Frazier. The interaction of syntax and semantics during sentence processing: Eye movements in the analysis of semantically biased sentences. Journal of verbal learning and verbal behavior, 22(3):358–374, 1983. [find](https://scholar.google.com/scholar?q=Keith+Rayner%2C+Marcia+Carlson%2C+and+Lyn+Frazier%2E+The+interaction+of+syntax+and+semantics+during+sentence+processing%3A+Eye+movements+in+the+analysis+of+semantically+biased+sentences%2E+Journal+of+verbal+lea)
40. Subhro Roy, Sam Thomson, Tongfei Chen, Richard Shin, Adam Pauls, Jason Eisner, and Benjamin Van Durme. BenchCLAMP: A benchmark for evaluating language models on semantic parsing. arXiv preprint arXiv:2206.10668, 2022. — [arXiv:2206.10668](https://arxiv.org/abs/2206.10668) [find](https://scholar.google.com/scholar?q=Subhro+Roy%2C+Sam+Thomson%2C+Tongfei+Chen%2C+Richard+Shin%2C+Adam+Pauls%2C+Jason+Eisner%2C+and+Benjamin+Van+Durme%2E+BenchCLAMP%3A+A+benchmark+for+evaluating+language+models+on+semantic+parsing%2E+arXiv+preprint+arXiv%3A)
41. Rachel Rudinger and Benjamin Van Durme. Is the stanford dependency representation semantic? In Proceedings of the Second Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pp. 54–58, 2014. [find](https://scholar.google.com/scholar?q=Rachel+Rudinger+and+Benjamin+Van+Durme%2E+Is+the+stanford+dependency+representation+semantic%3F+In+Proceedings+of+the+Second+Workshop+on+EVENTS%3A+Definition%2C+Detection%2C+Coreference%2C+and+Representation%2C+pp%2E)
42. Keisuke Sakaguchi and Benjamin Van Durme. Efficient online scalar annotation with bounded support. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 208–218, Melbourne, Australia, July 2018. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi+and+Benjamin+Van+Durme%2E+Efficient+online+scalar+annotation+with+bounded+support%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A)
43. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). arXiv preprint arXiv:2302.04761, 2023.
44. Hinrich Schutze. Ambiguity in language learning: Computational and cognitive models. Stanford University, 1995. [find](https://scholar.google.com/scholar?q=Hinrich+Schutze%2E+Ambiguity+in+language+learning%3A+Computational+and+cognitive+models%2E+Stanford+University%2C+1995%2E)
45. Semantic Machines, Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman,… — [Task-Oriented Dialogue as Dataflow Synthesis](https://md.research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/)
46. Richard Shin and Benjamin Van Durme. [Few-shot semantic parsing with language models trained on code](https://md.research.elamur.ai/free/en/papers/shin-2022-few-shot-semantic-parsing-language/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5417–5425, 2022.
47. Richard Shin, Christopher Lin, Sam Thomson, Charles Chen Jr, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Langu… [find](https://scholar.google.com/scholar?q=Richard+Shin%2C+Christopher+Lin%2C+Sam+Thomson%2C+Charles+Chen+Jr%2C+Subhro+Roy%2C+Emmanouil+Antonios+Platanios%2C+Adam+Pauls%2C+Dan+Klein%2C+Jason+Eisner%2C+and+Benjamin+Van+Durme%2E+Constrained+language+models+yield+fe)
48. Mark Steedman. Taking scope: The natural semantics of quantifiers. Mit Press, 2011. [find](https://scholar.google.com/scholar?q=Mark+Steedman%2E+Taking+scope%3A+The+natural+semantics+of+quantifiers%2E+Mit+Press%2C+2011%2E)
49. Elias Stengel-Eskin and Benjamin Van Durme. Calibrated interpretation: Confidence estimation in semantic parsing. arXiv preprint arXiv:2211.07443, 2022. [find](https://scholar.google.com/scholar?q=Elias+Stengel%2DEskin+and+Benjamin+Van+Durme%2E+Calibrated+interpretation%3A+Confidence+estimation+in+semantic+parsing%2E+arXiv+preprint+arXiv%3A2211%2E07443%2C+2022%2E)
50. Elias Stengel-Eskin and Benjamin Van Durme. Did you mean...? Confidence-based trade-offs in semantic parsing. arXiv preprint arXiv:2303.16857, 2023. [find](https://scholar.google.com/scholar?q=Elias+Stengel%2DEskin+and+Benjamin+Van+Durme%2E+Did+you+mean%2E%2E%2E%3F+Confidence%2Dbased+trade%2Doffs+in+semantic+parsing%2E+arXiv+preprint+arXiv%3A2303%2E16857%2C+2023%2E)
51. Elias Stengel-Eskin, Jimena Guallar-Blasco, Yi Zhou, and Benjamin Van Durme. Why did the chicken cross the road? Rephrasing and analyzing ambiguous questions in vqa. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023. [find](https://scholar.google.com/scholar?q=Elias+Stengel%2DEskin%2C+Jimena+Guallar%2DBlasco%2C+Yi+Zhou%2C+and+Benjamin+Van+Durme%2E+Why+did+the+chicken+cross+the+road%3F+Rephrasing+and+analyzing+ambiguous+questions+in+vqa%2E+Proceedings+of+the+61st+Annual+Mee)
52. Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew Walter, Ashis Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pp. 1507–1514, 2011. [find](https://scholar.google.com/scholar?q=Stefanie+Tellex%2C+Thomas+Kollar%2C+Steven+Dickerson%2C+Matthew+Walter%2C+Ashis+Banerjee%2C+Seth+Teller%2C+and+Nicholas+Roy%2E+Understanding+natural+language+commands+for+robotic+navigation+and+mobile+manipulation%2E)
53. Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3:25–55, 2020. [find](https://scholar.google.com/scholar?q=Stefanie+Tellex%2C+Nakul+Gopalan%2C+Hadas+Kress%2DGazit%2C+and+Cynthia+Matuszek%2E+Robots+that+use+language%2E+Annual+Review+of+Control%2C+Robotics%2C+and+Autonomous+Systems%2C+3%3A25%E2%80%9355%2C+2020%2E)
54. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv preprint arXiv:2302.13971, 2023.
55. Siddharth Vashishtha, Benjamin Van Durme, and Aaron Steven White. Fine-grained temporal relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2906–2919, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P… [find](https://scholar.google.com/scholar?q=Siddharth+Vashishtha%2C+Benjamin+Van+Durme%2C+and+Aaron+Steven+White%2E+Fine%2Dgrained+temporal+relation+extraction%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C)
56. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [Finetuned language models are zero-shot learners](https://md.research.elamur.ai/free/en/papers/wei-2021-finetuned-language-models-zero-shot/). In International Conference on Learning Representations, 2022.
57. Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. [find](https://scholar.google.com/scholar?q=Terry+Winograd%2E+Understanding+natural+language%2E+Cognitive+psychology%2C+3%281%29%3A1%E2%80%93191%2C+1972%2E)
58. Ludwig Wittgenstein. Tractatus Logico-Philosophicus. Annalen der Naturphilosophie, 1921. [find](https://scholar.google.com/scholar?q=Ludwig+Wittgenstein%2E+Tractatus+Logico%2DPhilosophicus%2E+Annalen+der+Naturphilosophie%2C+1921%2E)
59. Zhengxuan Wu, Christopher D Manning, and Christopher Potts. ReCOGS: How incidental details of a logical form overshadow an evaluation of semantic interpretation. arXiv preprint arXiv:2303.13716, 2023. [find](https://scholar.google.com/scholar?q=Zhengxuan+Wu%2C+Christopher+D+Manning%2C+and+Christopher+Potts%2E+ReCOGS%3A+How+incidental+details+of+a+logical+form+overshadow+an+evaluation+of+semantic+interpretation%2E+arXiv+preprint+arXiv%3A2303%2E13716%2C+2023%2E)
60. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018 Conference on Empirical Methods…
61. Yuewei Yuan, Chaitanya Malaviya, and Mark Yatskar. [AmbiCoref: Evaluating human and model sensitivity to ambiguous coreference](https://md.research.elamur.ai/free/en/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/). In Findings of the Association for Computational Linguistics: EACL 2023, pp. 993–1000, 2023.
62. John M Zelle and Raymond J Mooney. [Learning to parse database queries using inductive logic programming](https://md.research.elamur.ai/free/en/papers/zelle-1996-learning-parse-database-queries-using/). In Proceedings of the national conference on artificial intelligence, pp. 1050–1055, 1996.
63. Sheng Zhang, Xutai Ma, Kevin Duh, and Benjamin Van Durme. [Broad-coverage semantic parsing as transduction](https://md.research.elamur.ai/free/en/papers/zhang-2019-broad-coverage-semantic-parsing-as/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3786–3798…
64. George Kingsley Zipf. Human Behaviour and the Principle of Least Effort. Addison-Wesley Press, 1949. [find](https://scholar.google.com/scholar?q=George+Kingsley+Zipf%2E+Human+Behaviour+and+the+Principle+of+Least+Effort%2E+Addison%2DWesley+Press%2C+1949%2E)

*Updated 31 October 2026*