---
title: "L2CEval — paper by Ni et al. (2024) — summary and ideas"
description: "Large language models trained on code can turn natural-language requests into executable programs… 80 references, cited by 10 papers here."
authors: ["Ansong Ni", "Pengcheng Yin", "Yilun Zhao", "Martin Riddell", "Troy Feng", "Rui Shen", "Stephen Yin", "Ye Liu", "Semih Yavuz", "Caiming Xiong", "Shafiq Joty", "Yingbo Zhou", "Dragomir Radev", "Arman Cohan"]
year: 2024
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/2024.tacl-1.72/"
doi: "10.1162/tacl_a_00705"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ni-2024-l2ceval-evaluating-language-code-generation/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ni-2024-l2ceval-evaluating-language-code-generation/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/) / L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models

# L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models

**Authors:** [Ansong Ni](https://md.research.elamur.ai/free/en/authors/ansong-ni/), [Pengcheng Yin](https://md.research.elamur.ai/free/en/authors/pengcheng-yin/), [Yilun Zhao](https://md.research.elamur.ai/free/en/authors/yilun-zhao/), [Martin Riddell](https://md.research.elamur.ai/free/en/authors/martin-riddell/), Troy Feng, Rui Shen, Stephen Yin, [Ye Liu](https://md.research.elamur.ai/free/en/authors/ye-liu/), [Semih Yavuz](https://md.research.elamur.ai/free/en/authors/semih-yavuz/), [Caiming Xiong](https://md.research.elamur.ai/free/en/authors/caiming-xiong/), [Shafiq Joty](https://md.research.elamur.ai/free/en/authors/shafiq-joty/), [Yingbo Zhou](https://md.research.elamur.ai/free/en/authors/yingbo-zhou/), [Dragomir Radev](https://md.research.elamur.ai/free/en/authors/dragomir-radev/) and [Arman Cohan](https://md.research.elamur.ai/free/en/authors/arman-cohan/) · **Year:** 2024 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2024](https://md.research.elamur.ai/free/en/venues/tacl/2024/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2024.tacl-1.72/) · [PDF](https://aclanthology.org/2024.tacl-1.72.pdf) · [DOI 10.1162/tacl\_a\_00705](https://doi.org/10.1162/tacl_a_00705) · [OpenAlex](https://openalex.org/W4403296573)

**Citations:** 17 citations (OpenAlex, 2026-10-14) · [10](https://md.research.elamur.ai/free/en/papers/ni-2024-l2ceval-evaluating-language-code-generation/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Large language models trained on code can turn natural-language requests into executable programs, but earlier evaluations covered isolated tasks, single model families or particular learning setups. This fragmentation makes it hard to compare models or understand what drives performance. L2CEval addresses this gap with a unified benchmark covering 7 language-to-code tasks in three domains: text-to-SQL semantic parsing, math word problems solved through Python, and general Python programming. The authors test 56 models from 13 organizations, ranging from about 1B parameters up to Falcon-180B and OpenAI's davinci and GPT-4. To keep comparisons controlled, every model gets the same prompts, greedy decoding and execution-based accuracy. The study examines how results depend on model scale, pretraining data mix, instruction tuning and zero- vs. few-shot prompting. It also measures sensitivity to exemplars and confidence calibration, and uses human annotation to categorize typical errors. The evaluation code and all model outputs, including logits, are publicly released. This gives practitioners a reproducible reference for choosing models and supports future LLM development.

## Key points
- Seven datasets make up the benchmark: Spider (1,032) and WikiTQ (2,831) for SQL; GSM8k (1,495) and SVAMP (1,992) for math; MBPP (500), HumanEval (164) and DS-1000 (1,000) for Python.
- The evaluated models were pretrained on 35B to 3.5T tokens and include both open-source and proprietary, instruction-tuned and base variants. Scientific conclusions rely mainly on the open-source models.
- Correctness is judged by comparing execution results with gold outputs. Human checks found that a notable share of "correct" Spider programs were spurious, meaning they succeeded only by chance, while this was not observed for the other datasets.
- The authors acknowledge three limitations: only greedy decoding was assessed (no pass@k), cross-family comparisons are confounded by differences in architecture and objectives, and proprietary models disclose little about themselves.
- The framework and outputs are released at l2c-eval.github.io, which will host evaluations of newer models such as LLaMA-3 and StarCoder2.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 430 works · 424,363 citations
- [Allen Institute for Artificial Intelligence](https://md.research.elamur.ai/free/en/labs/allen-institute-for-artificial-intelligence/) — Research institute, Seattle, United States · 275 works · 97,634 citations
- [Salesforce](https://md.research.elamur.ai/free/en/companies/salesforce/) — Company, San Francisco, United States · 56 works · 17,867 citations
- [Google DeepMind](https://md.research.elamur.ai/free/en/companies/google-deepmind/) — Company, London, United Kingdom · 56 works · 78,880 citations
- [Yale University](https://md.research.elamur.ai/free/en/universities/yale-university/) — University, New Haven, United States · 95 works · 12,765 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 1,437 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 129 works

## Cite

Cite as: Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, Shafiq Joty, Yingbo Zhou, Dragomir Radev and Arman Cohan. 2024. L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{ni2024l2ceval,
  title = {{L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models}},
  author = {Ansong Ni, Pengcheng Yin, Yilun Zhao, Martin Riddell, Troy Feng, Rui Shen, Stephen Yin, Ye Liu, Semih Yavuz, Caiming Xiong, Shafiq Joty, Yingbo Zhou, Dragomir Radev and Arman Cohan},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00705},
  url = {https://doi.org/10.1162/tacl_a_00705}
}
```

- [← previous Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/chrysostomou-2024-investigating-hallucinations-pruned-large-language/)
- [next → LLM-as-a-Judge & Reward Model: What They Can and Cannot Do](https://md.research.elamur.ai/free/en/papers/son-2024-llm-judge-reward-model-they/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/7/)

## References

80 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Replit model. [https://huggingface.co/replit/replit-code-v1-3b](https://huggingface.co/replit/replit-code-v1-3b). Accessed: 2023-09-30. [↗](https://huggingface.co/replit/replit-code-v1-3b)
2. Katrin Affolter, Kurt Stockinger, and Abraham Bernstein. 2019. A comparative survey of recent natural language interfaces for databases. The VLDB Journal, 28:793–819. https://doi/10.1007/s00778-019-00567-8 [find](https://scholar.google.com/scholar?q=Katrin+Affolter%2C+Kurt+Stockinger%2C+and+Abraham+Bernstein%2E+2019%2E+A+comparative+survey+of+recent+natural+language+interfaces+for+databases%2E+The+VLDB+Journal%2C+28%3A793%E2%80%93819%2E+https%3A%2F%2Fdoi%2F10%2E1007%2Fs00778%2D019%2D00)
3. Rajas Agashe, Srinivasan Iyer, and Luke Zettlemoyer. 2019. [JuICe: A large scale distantly supervised dataset for open domain context-based code generation](https://md.research.elamur.ai/free/en/papers/agashe-2019-juice-large-scale-distantly-supervised/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
4. Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Rom… — [SantaCoder: don't reach for the stars!](https://md.research.elamur.ai/free/en/papers/allal-2023-santacoder-don-t-reach-stars/)
5. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Maitha Alhammadi, Mazzotta Daniele, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo. 2023. The falcon series of language models: Towards open fron… [find](https://scholar.google.com/scholar?q=Ebtesam+Almazrouei%2C+Hamza+Alobeidli%2C+Abdulaziz+Alshamsi%2C+Alessandro+Cappelli%2C+Ruxandra+Cojocaru%2C+Maitha+Alhammadi%2C+Mazzotta+Daniele%2C+Daniel+Heslow%2C+Julien+Launay%2C+Quentin+Malartic%2C+Badreddine+Noune%2C+B)
6. Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, et al. 2020. [Task-oriented dialogue as dataflow synthesis](https://md.research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/). Transactions of the Association for Computational Linguistics, 8:556–571. https://doi.org/10.1162 /tac…
7. Ion Androutsopoulos, Graeme D. Ritchie, and Peter Thanisch. 1995. Natural language interfaces to databases–an introduction. Natural Language Engineering, 1(1):29–81. https://doi.org/10.1017/S135132490000005X [↗](https://doi.org/10.1017/S135132490000005X)
8. Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan… [find](https://scholar.google.com/scholar?q=Ben+Athiwaratkun%2C+Sanjay+Krishna+Gouda%2C+Zijian+Wang%2C+Xiaopeng+Li%2C+Yuchen+Tian%2C+Ming+Tan%2C+Wasi+Uddin+Ahmad%2C+Shiqi+Wang%2C+Qing+Sun%2C+Mingyue+Shang%2C+Sujan+Kumar+Gonugondla%2C+Hantian+Ding%2C+Varun+Kumar%2C+Natha)
9. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
10. Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. 2022. A framework for the evaluation of code generation models. [https://github.com/bigcode-project/bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness). [↗](https://github.com/bigcode-project/bigcode-evaluation-harness)
11. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. [Semantic parsing on Freebase from question-answer pairs](https://md.research.elamur.ai/free/en/papers/berant-2013-semantic-parsing-freebase-question-answer/). In Empirical Methods in Natural Language Processing (EMNLP).
12. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conferenc… [find](https://scholar.google.com/scholar?q=Stella+Biderman%2C+Hailey+Schoelkopf%2C+Quentin+Gregory+Anthony%2C+Herbie+Bradley%2C+Kyle+O%E2%80%99Brien%2C+Eric+Hallahan%2C+Mohammad+Aflah+Khan%2C+Shivanshu+Purohit%2C+USVSN+Sai+Prashanth%2C+Edward+Raff%2C+et+al%2E+2023%2E+Pythia%3A)
13. Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022. [Gpt-neox-20b: An open-source autoregr…](https://md.research.elamur.ai/free/en/papers/black-2022-gpt-neox-20b-open-source/)
14. Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2023. Multipl-e: A scalable and polyglot approach to benchmarking neural code genera… [find](https://scholar.google.com/scholar?q=Federico+Cassano%2C+John+Gouwar%2C+Daniel+Nguyen%2C+Sydney+Nguyen%2C+Luna+Phipps%2DCostin%2C+Donald+Pinckney%2C+Ming%2DHo+Yee%2C+Yangtian+Zi%2C+Carolyn+Jane+Anderson%2C+Molly+Q%2E+Feldman%2C+Arjun+Guha%2C+Michael+Greenberg%2C+and+)
15. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
16. Zhoujun Cheng, Tianbao Xie, Peng Shi, Chengzu Li, Rahul Nadkarni, Yushi Hu, Caiming Xiong, Dragomir Radev, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. [Binding language models in symbolic languages](https://md.research.elamur.ai/free/en/papers/cheng-2022-binding-language-models-symbolic-languages/). ArXiv preprint arXiv:arXiv:2210.02875. Aakanksha Chowdhery, Sharan Narang, Jacob Devl…
17. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
18. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [find](https://scholar.google.com/scholar?q=Mike+Conover%2C+Matt+Hayes%2C+Ankit+Mathur%2C+Jianwei+Xie%2C+Jun+Wan%2C+Sam+Shah%2C+Ali+Ghodsi%2C+Patrick+Wendell%2C+Matei+Zaharia%2C+and+Reynold+Xin%2E+2023%2E+Free+dolly%3A+Introducing+the+world%E2%80%99s+first+truly+open+instruct)
19. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023. Incoder: A generative model for code infilling and synthesis. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Scott+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2023%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synthe)
20. Yao Fu, Hao Peng, and Tushar Khot. 2022. How does gpt obtain its ability? Tracing emergent abilities of language models to their sources. Yao Fu’s Notion. [find](https://scholar.google.com/scholar?q=Yao+Fu%2C+Hao+Peng%2C+and+Tushar+Khot%2E+2022%2E+How+does+gpt+obtain+its+ability%3F+Tracing+emergent+abilities+of+language+models+to+their+sources%2E+Yao+Fu%E2%80%99s+Notion%2E)
21. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Pal: Program-aided language models. In International Conference on Machine Learning, pages 10764–10799. PMLR. [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2023%2E+Pal%3A+Program%2Daided+language+models%2E+In+International+Conference+on+Machine+Learning%2C+page)
22. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321–1330. PMLR. [find](https://scholar.google.com/scholar?q=Chuan+Guo%2C+Geoff+Pleiss%2C+Yu+Sun%2C+and+Kilian+Q%2E+Weinberger%2E+2017%2E+On+calibration+of+modern+neural+networks%2E+In+International+Conference+on+Machine+Learning%2C+pages+1321%E2%80%931330%2E+PMLR%2E)
23. Rahul Gupta, Soham Pal, Aditya Kanade, and Shirish K. Shevade. 2017. DeepFix: Fixing common C language errors by deep learning. In AAAI Conference on Artificial Intelligence. https://doi.org/10.1609/aaai .v31i1.10742 [↗](https://doi.org/10.1609/aaai)
24. Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Luke Benson, Lucy Sun, Ekaterina Zubova, Yujie Qiao, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander R. Fabbri, Wojciech Kryscinski, Semih Y… — [FOLIO: Natural Language Reasoning with First-Order Logic](https://md.research.elamur.ai/free/en/papers/han-2022-folio-natural-language-reasoning-first/)
25. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. [Measuring coding challenge competence with apps](https://md.research.elamur.ai/free/en/papers/hendrycks-2021-measuring-coding-challenge-competence-apps/). arXiv preprint arXiv:2105.09938.
26. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — [Training Compute-Optimal Large Language Models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/)
27. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. 2023. Inner monologue: Embodied reasoning through planning wi… [find](https://scholar.google.com/scholar?q=Wenlong+Huang%2C+Fei+Xia%2C+Ted+Xiao%2C+Harris+Chan%2C+Jacky+Liang%2C+Pete+Florence%2C+Andy+Zeng%2C+Jonathan+Tompson%2C+Igor+Mordatch%2C+Yevgen+Chebotar%2C+Pierre+Sermanet%2C+Noah+Brown%2C+Tomas+Jackson%2C+Linda+Luu%2C+Sergey+Le)
28. Hamel Husain, Hongqi Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. [Codesearchnet challenge: Evaluating the state of semantic code search](https://md.research.elamur.ai/free/en/papers/husain-2020-codesearchnet-challenge-evaluating-state-semantic/). ArXiv, abs/1909.09436.
29. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping language to code in programmatic context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1643–1652. https://doi.org/10 .18653/v1/D18-1192 [find](https://scholar.google.com/scholar?q=Srinivasan+Iyer%2C+Ioannis+Konstas%2C+Alvin+Cheung%2C+and+Luke+Zettlemoyer%2E+2018%2E+Mapping+language+to+code+in+programmatic+context%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Lang)
30. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. [find](https://scholar.google.com/scholar?q=Ziwei+Ji%2C+Nayeon+Lee%2C+Rita+Frieske%2C+Tiezheng+Yu%2C+Dan+Su%2C+Yan+Xu%2C+Etsuko+Ishii%2C+Ye+Jin+Bang%2C+Andrea+Madotto%2C+and+Pascale+Fung%2E+2023%2E)
31. [Survey of hallucination in natural language generation](https://md.research.elamur.ai/free/en/papers/ji-2022-survey-hallucination-natural-language-generation/). ACM Computing Surveys, 55(12):1–38. https://doi.org/10.1145/3571730 Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccion… [↗](https://doi.org/10.1145/3571730)
32. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+Le%CC%81lio+Renard+La)
33. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. [Scaling laws for neural language models](https://md.research.elamur.ai/free/en/papers/kaplan-2020-scaling-laws-neural-language-models/). arXiv preprint arXiv:2001.08361.
34. Jens Krinke and Chaiyong Ragkhitwetsagul. 2022. Bigclonebench considered harmful for machine learning. 2022 IEEE 16th International Workshop on Software Clones (IWSC) pages 1–7. https://doi.org/10.1109/IWSC55060.2022.00008 [↗](https://doi.org/10.1109/IWSC55060.2022.00008)
35. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR. [find](https://scholar.google.com/scholar?q=Yuhang+Lai%2C+Chengxi+Li%2C+Yiming+Wang%2C+Tianyi+Zhang%2C+Ruiqi+Zhong%2C+Luke+Zettlemoyer%2C+Wen%2Dtau+Yih%2C+Daniel+Fried%2C+Sida+Wang%2C+and+Tao+Yu%2E+2023%2E+Ds%2D1000%3A+A+natural+and+reliable+benchmark+for+data+science+cod)
36. Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2024. [Can llm already serve as a database interface?](https://md.research.elamur.ai/free/en/papers/li-2023-llm-already-serve-database-interface/) A big bench for large-scale database…
37. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicola… — [arXiv:2206.02336](https://arxiv.org/abs/2206.02336) [find](https://scholar.google.com/scholar?q=Raymond+Li%2C+Loubna+Ben+Allal%2C+Yangtian+Zi%2C+Niklas+Muennighoff%2C+Denis+Kocetkov%2C+Chenghao+Mou%2C+Marc+Marone%2C+Christopher+Akiki%2C+Jia+Li%2C+Jenny+Chim%2C+Qian+Liu%2C+Evgenii+Zheltonozhskii%2C+Terry+Yue+Zhuo%2C+Thoma)
38. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James… — [Competition-Level Code Generation with AlphaCode](https://md.research.elamur.ai/free/en/papers/li-2022-competition-level-code-generation-alphacode/)
39. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… [find](https://scholar.google.com/scholar?q=Percy+Liang%2C+Rishi+Bommasani%2C+Tony+Lee%2C+Dimitris+Tsipras%2C+Dilara+Soylu%2C+Michihiro+Yasunaga%2C+Yian+Zhang%2C+Deepak+Narayanan%2C+Yuhuai+Wu%2C+Ananya+Kumar%2C+Benjamin+Newman%2C+Binhang+Yuan%2C+Bobby+Yan%2C+Ce+Zhang%2C+C)
40. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2023. Codexglue… [find](https://scholar.google.com/scholar?q=Shuai+Lu%2C+Daya+Guo%2C+Shuo+Ren%2C+Junjie+Huang%2C+Alexey+Svyatkovskiy%2C+Ambrosio+Blanco%2C+Colin+Clement%2C+Dawn+Drain%2C+Daxin+Jiang%2C+Duyu+Tang%2C+Ge+Li%2C+Lidong+Zhou%2C+Linjun+Shou%2C+Long+Zhou%2C+Michele+Tufano%2C+Ming+Go)
41. Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023. Wizardcoder: Empowering code large language models with evol-instruct. [find](https://scholar.google.com/scholar?q=Ziyang+Luo%2C+Can+Xu%2C+Pu+Zhao%2C+Qingfeng+Sun%2C+Xiubo+Geng%2C+Wenxiang+Hu%2C+Chongyang+Tao%2C+Jing+Ma%2C+Qingwei+Lin%2C+and+Daxin+Jiang%2E+2023%2E+Wizardcoder%3A+Empowering+code+large+language+models+with+evol%2Dinstruct%2E)
42. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and Ashwin Kalyan. 2022. [Lila: A unified benchmark for mathematical reasoning](https://md.research.elamur.ai/free/en/papers/mishra-2022-lila-unified-benchmark-mathematical-reasoning/). In Proceedings of the 2022 Conference on Empirical Methods in Natural…
43. Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29. https://doi.org/10.1609/aaai.v29i1.9602 [↗](https://doi.org/10.1609/aaai.v29i1.9602)
44. Ansong Ni, Jeevana Priya Inala, Chenglong Wang, Alex Polozov, Christopher Meek, Dragomir Radev, and Jianfeng Gao. 2023a. Learning math reasoning from self-sampled correct and partially-correct solutions. In The Eleventh International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Ansong+Ni%2C+Jeevana+Priya+Inala%2C+Chenglong+Wang%2C+Alex+Polozov%2C+Christopher+Meek%2C+Dragomir+Radev%2C+and+Jianfeng+Gao%2E+2023a%2E+Learning+math+reasoning+from+self%2Dsampled+correct+and+partially%2Dcorrect+solutio)
45. Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023b. [Lever: Learning to verify language-to-code generation with execution](https://md.research.elamur.ai/free/en/papers/ni-2023-lever-learning-verify-language-code/). In International Conference on Machine Learning, pages 26106–26128. PMLR.
46. Ansong Ni, Pengcheng Yin, and Graham Neubig. 2020. Merging weak and active supervision for semantic parsing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8536–8543. https://doi.org/10.1609/aaai.v34i05.6375 [↗](https://doi.org/10.1609/aaai.v34i05.6375)
47. Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou. 2023. Codegen2: Lessons for training llms on programming and natural languages. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Hiroaki+Hayashi%2C+Caiming+Xiong%2C+Silvio+Savarese%2C+and+Yingbo+Zhou%2E+2023%2E+Codegen2%3A+Lessons+for+training+llms+on+programming+and+natural+languages%2E)
48. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. arXiv preprint arXiv:2203.13474. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+A+conversational+paradigm+for+program+synthesis%2E+arXiv+preprint+arXiv%3A2203%2E13474%2E)
49. OpenAI. 2022. Chatgpt: Optimizing language models for dialogue. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2022%2E+Chatgpt%3A+Optimizing+language+models+for+dialogue%2E)
50. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
51. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. [Train…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
52. Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014. [find](https://scholar.google.com/scholar?q=Bhargavi+Paranjape%2C+Scott+Lundberg%2C+Sameer+Singh%2C+Hannaneh+Hajishirzi%2C+Luke+Zettlemoyer%2C+and+Marco+Tulio+Ribeiro%2E+2023%2E+Art%3A+Automatic+multi%2Dstep+reasoning+and+tool%2Duse+for+large+language+models%2E+arXi)
53. Panupong Pasupat and Percy Liang. 2015. [Compositional semantic parsing on semi-structured tables](https://md.research.elamur.ai/free/en/papers/pasupat-2015-compositional-semantic-parsing-semi-structured/). In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1…
54. Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. [Are nlp models really able to solve simple math word problems?](https://md.research.elamur.ai/free/en/papers/patel-2021-are-nlp-models-really-able/) In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080–2094. https://doi.org/…
55. Nitarshan Rajkumar, Raymond Li, and Dzmitry Bahdanau. 2022. Evaluating the text-to-sql capabilities of large language models. arXiv preprint arXiv:2204.00498. [find](https://scholar.google.com/scholar?q=Nitarshan+Rajkumar%2C+Raymond+Li%2C+and+Dzmitry+Bahdanau%2E+2022%2E+Evaluating+the+text%2Dto%2Dsql+capabilities+of+large+language+models%2E+arXiv+preprint+arXiv%3A2204%2E00498%2E)
56. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Fai… — [Code Llama: Open Foundation Models for Code](https://md.research.elamur.ai/free/en/papers/roziere-2024-code-llama-open-foundation-models/)
57. Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. [Unsupervised translation of programming languages](https://md.research.elamur.ai/free/en/papers/lachaux-2020-unsupervised-translation-programming-languages/). Advances in Neural Information Processing Systems, 33:20601–20611.
58. Timo Schick, Jane Dwivedi-Yu, Roberto Dessı̀, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. [Toolformer: Language models can teach themselves to use tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/). Advances in Neural Information Processing Systems, 36.
59. Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. Natural language to code translation with execution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3533–3546. https://doi.org/10.18653/v1/2022.emnlp-main.231 [↗](https://doi.org/10.18653/v1/2022.emnlp-main.231)
60. Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020. [Alfred: A benchmark for interpreting grounded instructions for everyday tasks](https://md.research.elamur.ai/free/en/papers/shridhar-2019-alfred-benchmark-interpreting-grounded-instructions/). In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10737…
61. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. [https://github.com/tatsu-lab/stanford](https://github.com/tatsu-lab/stanford) alpaca MosaicML NLP Team. 2023a. Introducing mpt-30b: Raising the bar fo… [↗](https://github.com/tatsu-lab/stanford)
62. MosaicML NLP Team. 2023b. Introducing mpt-7b: A new standard for open-source, commercially usable llms. Accessed: 2023-05-05. [find](https://scholar.google.com/scholar?q=MosaicML+NLP+Team%2E+2023b%2E+Introducing+mpt%2D7b%3A+A+new+standard+for+open%2Dsource%2C+commercially+usable+llms%2E+Accessed%3A+2023%2D05%2D05%2E)
63. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. [Llama: Open and efficient foundation language models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv…
64. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/)
65. Ben Wang and Aran Komatsuzaki. 2021. [GPT-J-6B: A 6 billion parameter autoregressive language model](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). [https://github.com](https://github.com) kingoflolz/mesh-transformer-jax Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves c… [↗](https://github.com)
66. Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. 2023. Execution-based evaluation for open-domain code generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1271–1290. https://doi.org/10 .18653/v1/2023.findings-emnlp.89 [find](https://scholar.google.com/scholar?q=Zhiruo+Wang%2C+Shuyan+Zhou%2C+Daniel+Fried%2C+and+Graham+Neubig%2E+2023%2E+Execution%2Dbased+evaluation+for+open%2Ddomain+code+generation%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+EMNLP+2023%2C+p)
67. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. [Emergent abilities of large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-emergent-abilities-large-language-models/). Transactions on M…
68. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022b. [Chain of thought prompting elicits reasoning in large language models](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). arXiv preprint arXiv:2201.11903.
69. Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2023. [Generating sequences by learning to self-correct](https://md.research.elamur.ai/free/en/papers/welleck-2022-generating-sequences-learning-self-correct/). In The Eleventh International Conference on Learning Representations.
70. Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, et al. 2022. [Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models](https://md.research.elamur.ai/free/en/papers/xie-2022-unifiedskg-unifying-multi-tasking-structured/). In Proceedings of the 2022 Conference…
71. Yiheng Xu, SU Hongjin, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu. 2023. Lemur: Harmonizing natural language and code for language agents. In The Twelfth International Conf… [find](https://scholar.google.com/scholar?q=Yiheng+Xu%2C+SU+Hongjin%2C+Chen+Xing%2C+Boyu+Mi%2C+Qian+Liu%2C+Weijia+Shi%2C+Binyuan+Hui%2C+Fan+Zhou%2C+Yitao+Liu%2C+Tianbao+Xie%2C+Zhoujun+Cheng%2C+Siheng+Zhao%2C+Lingpeng+Kong%2C+Bailin+Wang%2C+Caiming+Xiong%2C+and+Tao+Yu%2E+2023%2E)
72. Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. [Learning to mine aligned code and natural language pairs from stack overflow](https://md.research.elamur.ai/free/en/papers/yin-2018-learning-mine-aligned-code-natural/). 2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR), pages 476–486. https://doi.org/10.1145/3196398.3196408 [↗](https://doi.org/10.1145/3196398.3196408)
73. Pengcheng Yin and Graham Neubig. 2017. [A syntactic neural model for general-purpose code generation](https://md.research.elamur.ai/free/en/papers/yin-2017-syntactic-neural-model-general-purpose/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440–450. Vancouver, Canada. Association for Computational Linguistics.
74. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. [Tabert: Pretraining for joint understanding of textual and tabular data](https://md.research.elamur.ai/free/en/papers/yin-2020-tabert-pretraining-joint-understanding-textual/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413–8426. https://doi.org/10.18653/v1/2020.acl-main.745 [↗](https://doi.org/10.18653/v1/2020.acl-main.745)
75. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. [Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018…
76. Luke S. Zettlemoyer and Michael Collins. 2005. [Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars](https://md.research.elamur.ai/free/en/papers/zettlemoyer-2012-learning-map-sentences-logical-form/). In UAI ’05, Proceedings of the 21st Conference in Uncertainty in Artificial Intelligence, Edinburgh, Scotland, July 26–29, 2005, pages 658–666.…
77. Tianyi Zhang, Tao Yu, Tatsunori Hashimoto, Mike Lewis, Wen-tau Yih, Daniel Fried, and Sida Wang. 2023. Coder reviewer reranking for code generation. In International Conference on Machine Learning, pages 41832–41846. PMLR. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Tao+Yu%2C+Tatsunori+Hashimoto%2C+Mike+Lewis%2C+Wen%2Dtau+Yih%2C+Daniel+Fried%2C+and+Sida+Wang%2E+2023%2E+Coder+reviewer+reranking+for+code+generation%2E+In+International+Conference+on+Machine+Learning%2C+pa)
78. Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shanshan Wang, Yufei Xue, Zi-Yuan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023. [Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x](https://md.research.elamur.ai/free/en/papers/zheng-2023-codegeex-pre-trained-model-code/). ArXiv, abs/2303.17568.
79. Ruiqi Zhong, Tao Yu, and Dan Klein. 2020. Semantic evaluation for text-to-sql with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 396–411. https://doi.org/10 .18653/v1/2020.emnlp-main.29 [find](https://scholar.google.com/scholar?q=Ruiqi+Zhong%2C+Tao+Yu%2C+and+Dan+Klein%2E+2020%2E+Semantic+evaluation+for+text%2Dto%2Dsql+with+distilled+test+suites%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMN)
80. Shuyan Zhou, Pengcheng Yin, and Graham Neubig. 2022. Hierarchical control of situated agents through natural language. In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pages 67–84. https://doi.org/10.18653/v1/2022.suki-1.8 [↗](https://doi.org/10.18653/v1/2022.suki-1.8)

*Updated 31 October 2026*