---
title: "Few-Shot Semantic Parsing with Language Models Trained on"
description: "Here the authors replace GPT-3 with OpenAI Codex (Cushman Codex, Davinci Codex) and test on… 14 references, cited by 36 papers here."
authors: ["Richard Shin", "Benjamin Van Durme"]
year: 2022
venue: "Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2022.naacl-main.396/"
doi: "10.18653/v1/2022.naacl-main.396"
arxiv: "2112.08696"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shin-2022-few-shot-semantic-parsing-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shin-2022-few-shot-semantic-parsing-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Few-Shot Semantic Parsing with Language Models Trained on Code

# Few-Shot Semantic Parsing with Language Models Trained on Code

**Authors:** [Richard Shin](https://md.research.elamur.ai/free/en/authors/richard-shin/) and [Benjamin Van Durme](https://md.research.elamur.ai/free/en/authors/benjamin-van-durme/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2022](https://md.research.elamur.ai/free/en/venues/naacl/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.naacl-main.396/) · [PDF](https://aclanthology.org/2022.naacl-main.396.pdf) · [DOI 10.18653/v1/2022.naacl-main.396](https://doi.org/10.18653/v1/2022.naacl-main.396) · [arXiv 2112.08696](https://arxiv.org/abs/2112.08696) · [OpenAlex](https://openalex.org/W4226226396)

**Citations:** 36 citations (OpenAlex, 2026-10-14) · [36](https://md.research.elamur.ai/free/en/papers/shin-2022-few-shot-semantic-parsing-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
This paper asks whether language models pre-trained partly on source code are better few-shot semantic parsers than models trained mainly on prose. The authors revisit the setup of Shin et al. (2021). That setup used GPT-3 with retrieved in-context examples (20 similar training pairs per query) and grammar-constrained decoding, and it produced either canonical utterances (a controlled English-like sublanguage mapped to programs via an SCFG) or the meaning representation itself. Here the authors replace GPT-3 with OpenAI Codex (Cushman Codex, Davinci Codex) and test on Overnight and SMCalFlow using only a few hundred training examples. They also adapt the Synchromesh idea into speculative constrained decoding, which reduces the number of API calls compared with beam search.

Codex beats comparable GPT-3 models. More importantly, it loses almost no accuracy when it outputs the Lisp-like meaning representations directly. This matters because hand-crafting canonical-utterance grammars is costly, so code-trained models make few-shot parsers cheaper to build.

## Key points
- **Data:** The experiments draw 200 training examples per Overnight domain (8 domains) and a stratified 300-example sample from SMCalFlow's 133,821 training pairs. Overnight is scored by denotation accuracy and SMCalFlow by syntactic match.
- **Canonical vs. meaning output on Overnight Calendar:**
  - Davinci drops from 0.81 to 0.68 when it targets meaning representations.
  - Davinci Codex scores 0.86 on both.
  - On SMCalFlow, the Codex gap is only 0.010.
- **Constraints and beam search still matter:** Even with Codex, constrained decoding and beam search help. With constraints and beam 5, Codex reaches 0.86 / 0.345, compared with 0.73 / 0.280 for unconstrained greedy decoding.
- **Speculative constrained decoding:** With width 5–10, it matches beam-search accuracy at notably higher throughput.
- **Full evaluation sets:** Codex reaches 0.785 (Overnight average) and 0.342 (SMCalFlow) with canonical utterances, and 0.750 / 0.330 with meaning representations. This surpasses the earlier GPT-3 results of 0.765 / 0.32.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧾 Semantic parsing & meaning representation](https://md.research.elamur.ai/free/en/topics/semantic-parsing/) · 185 works

## Cite

Cite as: Richard Shin and Benjamin Van Durme. 2022. Few-Shot Semantic Parsing with Language Models Trained on Code. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shin2022few,
  title = {{Few-Shot Semantic Parsing with Language Models Trained on Code}},
  author = {Richard Shin and Benjamin {Van Durme}},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.396},
  eprint = {2112.08696},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.396}
}
```

- [← previous Arabic Natural Language Processing](https://md.research.elamur.ai/free/en/papers/habash-2022-arabic-natural-language-processing/)
- [next → Infusing Knowledge from Wikipedia to Enhance Stance Detection](https://md.research.elamur.ai/free/en/papers/he-2022-infusing-knowledge-wikipedia-enhance-stance/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Chris… — [Task-Oriented Dialogue as Dataflow Synthesis](https://md.research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/)
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). CoRR, abs/2108.07732.
3. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610623, New York, NY, USA. Associati… [find](https://scholar.google.com/scholar?q=Emily+M%2E+Bender%2C+Timnit+Gebru%2C+Angelina+McMillan%2DMajor%2C+and+Shmargaret+Shmitchell%2E+2021%2E+On+the+dangers+of+stochastic+parrots%3A+Can+language+models+be+too+big%3F+In+Proceedings+of+the+2021+ACM+Conference)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mik… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde+de+Oliveira+Pinto%2C+Jared+Kaplan%2C+Harrison+Edwards%2C+Yuri+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C+M)
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — [PaLM: Scaling Language Modeling with Pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/)
7. Paresh Kharya and Ali Alvi. 2021. Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, the Worlds Largest and Most Powerful Generative Language Model. [https://developer.nvid](https://developer.nvid) ia.com/blog/using-deepspeed-and-me gatron-to-train-megatron-turing-nl g-530b-the-worlds-largest-and-most -powerful-… [↗](https://developer.nvid)
8. Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable semantic parsing via retrieval augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7683–7698, Online and Punta Cana, Dominican Republic. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Panupong+Pasupat%2C+Yuan+Zhang%2C+and+Kelvin+Guu%2E+2021%2E+Controllable+semantic+parsing+via+retrieval+augmentation%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)
9. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Chris Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable code generation from pre-trained language models. In ICLR 2022. [find](https://scholar.google.com/scholar?q=Gabriel+Poesia%2C+Alex+Polozov%2C+Vu+Le%2C+Ashish+Tiwari%2C+Gustavo+Soares%2C+Chris+Meek%2C+and+Sumit+Gulwani%2E+2022%2E+Synchromesh%3A+Reliable+code+generation+from+pre%2Dtrained+language+models%2E+In+ICLR+2022%2E)
10. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… [find](https://scholar.google.com/scholar?q=Jack+W%2E+Rae%2C+Sebastian+Borgeaud%2C+Trevor+Cai%2C+Katie+Millican%2C+Jordan+Hoffmann%2C+H%2E+Francis+Song%2C+John+Aslanides%2C+Sarah+Henderson%2C+Roman+Ring%2C+Susannah+Young%2C+Eliza+Rutherford%2C+Tom+Hennigan%2C+Jacob+Menick)
11. Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901, Online and Punta Cana, Dominican Repu… [find](https://scholar.google.com/scholar?q=Torsten+Scholak%2C+Nathan+Schucher%2C+and+Dzmitry+Bahdanau%2E+2021%2E+PICARD%3A+Parsing+incrementally+for+constrained+auto%2Dregressive+decoding+from+language+models%2E+In+Proceedings+of+the+2021+Conference+on+Empi)
12. Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios, Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natural La… [find](https://scholar.google.com/scholar?q=Richard+Shin%2C+Christopher+Lin%2C+Sam+Thomson%2C+Charles+Chen%2C+Subhro+Roy%2C+Emmanouil+Antonios+Platanios%2C+Adam+Pauls%2C+Dan+Klein%2C+Jason+Eisner%2C+and+Benjamin+Van+Durme%2E+2021%2E+Constrained+language+models+yield)
13. Ben Wang and Aran Komatsuzaki. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh](https://github.com/kingoflolz/mesh) -transformer-jax. [↗](https://github.com/kingoflolz/mesh)
14. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332–1342, Bei… [find](https://scholar.google.com/scholar?q=Yushi+Wang%2C+Jonathan+Berant%2C+and+Percy+Liang%2E+2015%2E+Building+a+semantic+parser+overnight%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+7th+Interna)

*Updated 29 October 2026*