---
title: "Natural Language to Code Translation with Execution — Shi 2022"
description: "This paper proposes MBR-EXEC, a selection method based on minimum Bayes risk. 39 references, cited by 70 papers here."
authors: ["Freda Shi", "Daniel Fried", "Marjan Ghazvininejad", "Luke Zettlemoyer", "Sida I. Wang"]
year: 2022
venue: "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2022.emnlp-main.231/"
doi: "10.18653/v1/2022.emnlp-main.231"
arxiv: "2204.11454"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/shi-2022-natural-language-code-translation-execution/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/shi-2022-natural-language-code-translation-execution/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Natural Language to Code Translation with Execution

# Natural Language to Code Translation with Execution

**Authors:** [Freda Shi](https://md.research.elamur.ai/free/en/authors/freda-shi/), [Daniel Fried](https://md.research.elamur.ai/free/en/authors/daniel-fried/), [Marjan Ghazvininejad](https://md.research.elamur.ai/free/en/authors/marjan-ghazvininejad/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) and [Sida I. Wang](https://md.research.elamur.ai/free/en/authors/sida-wang/) · **Year:** 2022 · **Venue:** [Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2022.emnlp-main.231/) · [PDF](https://aclanthology.org/2022.emnlp-main.231.pdf) · [DOI 10.18653/v1/2022.emnlp-main.231](https://doi.org/10.18653/v1/2022.emnlp-main.231) · [arXiv 2204.11454](https://arxiv.org/abs/2204.11454) · [OpenAlex](https://openalex.org/W4385572707)

**Citations:** 46 citations (OpenAlex, 2026-10-14) · [70](https://md.research.elamur.ai/free/en/papers/shi-2022-natural-language-code-translation-execution/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large pretrained code models such as Codex can often generate a correct program among many samples, but it is hard to pick the single right one. This paper proposes MBR-EXEC, a selection method based on minimum Bayes risk. For a natural-language request, the authors draw multiple candidate programs from Codex using few-shot prompts with a generic markup template. They run every candidate on a small set of test inputs, typically one. Programs that produce identical outputs are treated as semantically equivalent. The method then picks the candidate that agrees with the most other samples, using model likelihood to break ties. It never needs ground-truth outputs, and the model is not retrained. On MBPP (Python), Spider (SQL) and NL2Bash (Bash), MBR-EXEC beats every selection strategy that ignores execution. When code cannot be run, MBR with a BLEU-based risk is a reasonable fallback. These findings show that cheap, inference-time execution signals can substantially improve natural-language-to-code systems built on frozen pretrained models.

## Key points
- The loss is a 0/1 comparison of execution outputs on test inputs. A program that crashes is counted as matching no other candidate, and ties are resolved by highest likelihood.
- Approximate figures from Table 2 compare greedy decoding, plain sampling and MBR-EXEC:
  - MBPP: about 47.3 / 47.7 / 58.2
  - Spider: about 50.8 / 48.5 / 63.6
  - NL2Bash (char. BLEU): about 52.8 / 53.0 / 58.5
- Running several different 3-shot prompts drawn from 15 examples worked better than concatenating all 15 into one prompt.
- In an ablation, filtering candidates for executability alone helped all baselines considerably. On Spider, maximum-likelihood selection among executable queries even beat MBR-EXEC.
- Using two or three test cases improved over using one. A soft (fractional) loss performed about the same as the hard loss.
- Oracle Pass@K upper bounds exceeded supervised state-of-the-art results on all three datasets, which points to inference methods as a promising research direction.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations
- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 402 works · 212,763 citations
- [Toyota Technological Institute at Chicago](https://md.research.elamur.ai/free/en/universities/toyota-technological-institute-at-chicago/) — University, Chicago, United States · 35 works · 13,377 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,730 works

## Cite

Cite as: Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer and Sida I. Wang. 2022. Natural Language to Code Translation with Execution. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{shi2022natural,
  title = {{Natural Language to Code Translation with Execution}},
  author = {Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer and Sida I. Wang},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.231},
  eprint = {2204.11454},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.231}
}
```

- [← previous KG-FiD: Infusing Knowledge Graph in Fusion-in-Decoder for Open-Domain Question Answering](https://md.research.elamur.ai/free/en/papers/yu-2022-kg-fid-infusing-knowledge-graph/)
- [next → Calibrating Factual Knowledge in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/dong-2022-calibrating-factual-knowledge-pretrained-language/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/)

## References

39 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
2. Peter J Bickel and Kjell A Doksum. 1977. Mathematical statistics: basic ideas and selected topics, volumes I-II package. HoldenDay Inc., Oakland, CA, USA. [find](https://scholar.google.com/scholar?q=Peter+J+Bickel+and+Kjell+A+Doksum%2E+1977%2E+Mathematical+statistics%3A+basic+ideas+and+selected+topics%2C+volumes+I%2DII+package%2E+HoldenDay+Inc%2E%2C+Oakland%2C+CA%2C+USA%2E)
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
5. Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731–742, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Li+Dong+and+Mirella+Lapata%2E+2018%2E+Coarse%2Dto%2Dfine+decoding+for+neural+semantic+parsing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volume+1%3A+Long+Papers)
6. Bryan Eikema and Wilker Aziz. 2020. [Is MAP decoding all you need? the inadequacy of the mode in neural machine translation](https://md.research.elamur.ai/free/en/papers/eikema-2020-is-map-decoding-all-you/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 4506–4520, Barcelona, Spain (Online). International Committee on Computational Linguisti…
7. Bryan Eikema and Wilker Aziz. 2021. Sampling-based minimum bayes risk decoding for neural machine translation. arXiv preprint arXiv:2108.04718. — [arXiv:2108.04718](https://arxiv.org/abs/2108.04718) [find](https://scholar.google.com/scholar?q=Bryan+Eikema+and+Wilker+Aziz%2E+2021%2E+Sampling%2Dbased+minimum+bayes+risk+decoding+for+neural+machine+translation%2E+arXiv+preprint+arXiv%3A2108%2E04718%2E+%E2%80%94+%5BarXiv%3A2108%2E04718%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2108%2E04718%29)
8. Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. [Making pre-trained language models better few-shot learners](https://md.research.elamur.ai/free/en/papers/gao-2021-making-pre-trained-language-models/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pa…
9. Vaibhava Goel and William J Byrne. 2000. Minimum bayes-risk automatic speech recognition. Computer Speech & Language, 14(2):115–135. [find](https://scholar.google.com/scholar?q=Vaibhava+Goel+and+William+J+Byrne%2E+2000%2E+Minimum+bayes%2Drisk+automatic+speech+recognition%2E+Computer+Speech+%26+Language%2C+14%282%29%3A115%E2%80%93135%2E)
10. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring coding challenge competence with APPS](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). In Thirty-fifth Conference on Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubi…
11. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. [UNIFIEDQA: Crossing format boundaries with a single QA system](https://md.research.elamur.ai/free/en/papers/khashabi-2020-unifiedqa-crossing-format-boundaries-single/). In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1896–1907, Online. Association for Com…
12. Shankar Kumar and William Byrne. 2004. [Minimum Bayes-risk decoding for statistical machine translation](https://md.research.elamur.ai/free/en/papers/kumar-2004-minimum-bayes-risk-decoding-statistical/). In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176, Boston, Massachusetts, USA…
13. Marie-Anne Lachaux, Baptiste Roziere, Marc Szafraniec, and Guillaume Lample. 2021. Dobf: A deobfuscation pre-training objective for programming languages. Advances in Neural Information Processing Systems, 34. [find](https://scholar.google.com/scholar?q=Marie%2DAnne+Lachaux%2C+Baptiste+Roziere%2C+Marc+Szafraniec%2C+and+Guillaume+Lample%2E+2021%2E+Dobf%3A+A+deobfuscation+pre%2Dtraining+objective+for+programming+languages%2E+Advances+in+Neural+Information+Processing+Sys)
14. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. [The power of scale for parameter-efficient prompt tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguist…
15. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… [find](https://scholar.google.com/scholar?q=Xiang+Lisa+Li+and+Percy+Liang%2E+2021%2E+Prefix%2Dtuning%3A+Optimizing+continuous+prompts+for+generation%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11t)
16. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. [Competition-level code generation with alphacode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/). arXiv preprint arXiv:2203.07814.
17. Xi Victoria Lin, Chenglong Wang, Luke Zettlemoyer, and Michael D. Ernst. 2018. NL2Bash: A corpus and semantic parser for natural language interface to the linux operating system. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan… [find](https://scholar.google.com/scholar?q=Xi+Victoria+Lin%2C+Chenglong+Wang%2C+Luke+Zettlemoyer%2C+and+Michael+D%2E+Ernst%2E+2018%2E+NL2Bash%3A+A+corpus+and+semantic+parser+for+natural+language+interface+to+the+linux+operating+system%2E+In+Proceedings+of+the)
18. Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, Fumin Wang, and Andrew Senior. 2016. [Latent predictor networks for code generation](https://md.research.elamur.ai/free/en/papers/ling-2016-latent-predictor-networks-code-generation/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 599–609, Be…
19. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. [Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing](https://md.research.elamur.ai/free/en/papers/liu-2021-pre-train-prompt-predict-systematic/). arXiv preprint arXiv:2107.13586.
20. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, MING GONG, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie LIU. 2021. CodeXGLUE… [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+Ge+Li%2C+Lidong+Zhou%2C+Linjun+Shou%2C+Long+Zhou%2C+Michele+Tufano%2C+MING+GO)
21. Antonio Valerio Miceli Barone and Rico Sennrich. 2017. A parallel corpus of python functions and documentation strings for automated code documentation and code generation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 314–… [find](https://scholar.google.com/scholar?q=Antonio+Valerio+Miceli+Barone+and+Rico+Sennrich%2E+2017%2E+A+parallel+corpus+of+python+functions+and+documentation+strings+for+automated+code+documentation+and+code+generation%2E+In+Proceedings+of+the+Eight)
22. Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021. [Noisy channel language model prompting for few-shot text classification](https://md.research.elamur.ai/free/en/papers/min-2021-noisy-channel-language-model-prompting/). arXiv preprint arXiv:2108.04106.
23. Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1139– 1149, Vancouver, Canada. Association for Computati… [find](https://scholar.google.com/scholar?q=Maxim+Rabinovich%2C+Mitchell+Stern%2C+and+Dan+Klein%2E+2017%2E+Abstract+syntax+networks+for+code+generation+and+semantic+parsing%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computational)
24. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
25. Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901, Online and Punta Cana, Dominican Repu… [find](https://scholar.google.com/scholar?q=Torsten+Scholak%2C+Nathan+Schucher%2C+and+Dzmitry+Bahdanau%2E+2021%2E+PICARD%3A+Parsing+incrementally+for+constrained+auto%2Dregressive+decoding+from+language+models%2E+In+Proceedings+of+the+2021+Conference+on+Empi)
26. Haoyue Shi, Jiayuan Mao, Kevin Gimpel, and Karen Livescu. 2019. Visually grounded neural syntax acquisition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1842–1861, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Haoyue+Shi%2C+Jiayuan+Mao%2C+Kevin+Gimpel%2C+and+Karen+Livescu%2E+2019%2E+Visually+grounded+neural+syntax+acquisition%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C)
27. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. [AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts](https://md.research.elamur.ai/free/en/papers/shin-2020-autoprompt-eliciting-knowledge-language-models/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222–4235, Onlin…
28. Alane Suhr, Srinivasan Iyer, and Yoav Artzi. 2018. Learning to map context-dependent sentences to executable formal queries. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), page… [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Srinivasan+Iyer%2C+and+Yoav+Artzi%2E+2018%2E+Learning+to+map+context%2Ddependent+sentences+to+executable+formal+queries%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chapter+of+the+)
29. Ivan Titov and James Henderson. 2006. Bayes risk minimization in natural language parsing. University of Geneva technical report. [find](https://scholar.google.com/scholar?q=Ivan+Titov+and+James+Henderson%2E+2006%2E+Bayes+risk+minimization+in+natural+language+parsing%2E+University+of+Geneva+technical+report%2E)
30. Roy Tromble, Shankar Kumar, Franz Och, and Wolfgang Macherey. 2008. [Lattice Minimum Bayes-Risk decoding for statistical machine translation](https://md.research.elamur.ai/free/en/papers/tromble-2008-lattice-minimum-bayes-risk-decoding/). In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 620–629, Honolulu, Hawaii. Association for Computational Lingu…
31. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). Advances in neural information processing systems, 30.
32. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. [Self-consistency improves chain of thought reasoning in language models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). arXiv preprint arXiv:2203.11171.
33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). arXiv preprint arXiv:2201.11903.
34. Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020. [Neural text generation with unlikelihood training](https://md.research.elamur.ai/free/en/papers/welleck-2019-neural-text-generation-unlikelihood-training/). In International Conference on Learning Representations.
35. Chunyang Xiao, Marc Dymetman, and Claire Gardent. 2016. Sequence-based structured prediction for semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1341–1350, Berlin, Germany. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Chunyang+Xiao%2C+Marc+Dymetman%2C+and+Claire+Gardent%2E+2016%2E+Sequence%2Dbased+structured+prediction+for+semantic+parsing%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Lingui)
36. Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. 2022. [Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models](https://md.research.elamur.ai/free/en/papers/xie-2022-unifiedskg-unifying-multi-tasking-structured/). arXiv preprint arXiv:2201.05966.
37. Frank F. Xu, Zhengbao Jiang, Pengcheng Yin, Bogdan Vasilescu, and Graham Neubig. 2020. Incorporating external knowledge through pre-training for natural language to code generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6045–6052, Online. A… [find](https://scholar.google.com/scholar?q=Frank+F%2E+Xu%2C+Zhengbao+Jiang%2C+Pengcheng+Yin%2C+Bogdan+Vasilescu%2C+and+Graham+Neubig%2E+2020%2E+Incorporating+external+knowledge+through+pre%2Dtraining+for+natural+language+to+code+generation%2E+In+Proceedings+of+)
38. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. [Learning to mine aligned code and natural language pairs from stack overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/). In 2018 IEEE/ACM 15th international conference on mining software repositories (MSR), pages 476–486. IEEE.
39. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018…

*Updated 2 November 2026*