---
title: "Language Models of Code are Few-Shot Commonsense Learners"
description: "The authors propose CoCoGen, which rewrites each (text, graph) example as an ordinary Python class. 38 references, cited by 98 papers here."
authors: ["Aman Madaan", "Shuyan Zhou", "Uri Alon", "Yiming Yang", "Graham Neubig"]
year: 2022
venue: "arXiv:2210.07128"
source: "https://arxiv.org/abs/2210.07128"
doi: "10.48550/arxiv.2210.07128"
arxiv: "2210.07128"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/madaan-2022-language-models-code-few-shot/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/madaan-2022-language-models-code-few-shot/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/) / Language Models of Code are Few-Shot Commonsense Learners

# Language Models of Code are Few-Shot Commonsense Learners

**Authors:** [Aman Madaan](https://md.research.elamur.ai/free/en/authors/aman-madaan/), [Shuyan Zhou](https://md.research.elamur.ai/free/en/authors/shuyan-zhou/), [Uri Alon](https://md.research.elamur.ai/free/en/authors/uri-alon/), [Yiming Yang](https://md.research.elamur.ai/free/en/authors/yiming-yang/) and [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) · **Year:** 2022 · **Venue:** [arXiv:2210.07128](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2022](https://md.research.elamur.ai/free/en/venues/emnlp/2022/)) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2210.07128) · [PDF](https://arxiv.org/pdf/2210.07128) · [DOI 10.48550/arxiv.2210.07128](https://doi.org/10.48550/arxiv.2210.07128) · [arXiv 2210.07128](https://arxiv.org/abs/2210.07128) · [OpenAlex](https://openalex.org/W4385567216)

**Citations:** 114 citations (OpenAlex, 2026-10-14) · [98](https://md.research.elamur.ai/free/en/papers/madaan-2022-language-models-code-few-shot/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Structured commonsense reasoning asks a model to turn a natural-language input, such as a goal, into a graph, for example a script of steps or an explanation graph. Earlier approaches gave these graphs to language models as flattened edge lists or DOT strings. Those formats rarely appear in pretraining text and place connected nodes far apart, so models often produced malformed or inconsistent structures. The authors propose CoCoGen, which rewrites each (text, graph) example as an ordinary Python class. The input becomes a class attribute or comment, nodes become objects, and edges become attributes such as `children`. Several such classes form a few-shot prompt. A code model (CODEX) then completes a partial class for a new input, and the output is parsed back into a graph and scored with each dataset's standard metrics. Across PROSCRIPT, PROPARA and EXPLAGRAPHS, this setup outperforms GPT-3 models prompted with flattened text and also T5 models fine-tuned on full training data. The work shows that matching the output format to a model's pretraining data can strongly improve structured generation.

## Key points
- **Models compared:** CoCoGen uses `code-davinci-002` (CODEX) in few-shot mode. Baselines are the GPT-3 models CURIE (`text-curie-001`) and DAVINCI (`text-davinci-002`), plus fine-tuned T5-large (EXPLAGRAPHS) and T5-xxl results from Sakaguchi et al. (PROSCRIPT).
- **Tasks:** script generation (PROSCRIPT), entity state tracking (PROPARA) and explanation graph generation (EXPLAGRAPHS).
- **Prompts and output:** each prompt holds about 5–10 Python classes. CODEX almost always returned syntactically valid Python, which made converting its output back into graphs easy.
- **Ablations and evaluation:** the ablations cover code format, model size and the number of examples, and simpler class formats worked best with larger models. Human judgments agreed with the automatic metrics.
- **Release and limits:** code and data are available at github.com/madaan/CoCoGen. Limitations include reliance on closed OpenAI models and English-only datasets.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 598 works · 191,226 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 3,486 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,685 works

## Cite

Cite as: Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang and Graham Neubig. 2022. Language Models of Code are Few-Shot Commonsense Learners. arXiv:2210.07128.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{madaan2022language,
  title = {{Language Models of Code are Few-Shot Commonsense Learners}},
  author = {Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang and Graham Neubig},
  year = {2022},
  doi = {10.48550/arxiv.2210.07128},
  eprint = {2210.07128},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2210.07128}
}
```

- [← previous Demystifying Prompts in Language Models via Perplexity Estimation](https://md.research.elamur.ai/free/en/papers/gonen-2022-demystifying-prompts-language-models-perplexity/)
- [next → An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/meade-2022-empirical-survey-effectiveness-debiasing-techniques/)
- [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/2/)

## References

38 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Zeina Abu-Aisheh, Romain Raveaux, Jean-Yves Ramel, and Patrick Martineau. 2015. An exact graph edit distance algorithm for solving pattern recognition problems. In An exact graph edit distance algorithm for solving pattern recognition problems. [find](https://scholar.google.com/scholar?q=Zeina+Abu%2DAisheh%2C+Romain+Raveaux%2C+Jean%2DYves+Ramel%2C+and+Patrick+Martineau%2E+2015%2E+An+exact+graph+edit+distance+algorithm+for+solving+pattern+recognition+problems%2E+In+An+exact+graph+edit+distance+algorit)
2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. [Program synthesis with large language models](https://md.research.elamur.ai/free/en/papers/austin-2021-program-synthesis-large-language-models/). arXiv preprint arXiv:2108.07732.
3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — [Language Models are Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
4. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — [Evaluating Large Language Models Trained on Code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harri Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, et al. 2021b. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. [Training verifiers to solve math word problems](https://md.research.elamur.ai/free/en/papers/cobbe-2021-training-verifiers-solve-math-word/). arXiv preprint arXiv:2110.14168.
8. Alain Colmerauer and Philippe Roussel. 1996. The birth of prolog. In History of programming languages—II, pages 331–367. [find](https://scholar.google.com/scholar?q=Alain+Colmerauer+and+Philippe+Roussel%2E+1996%2E+The+birth+of+prolog%2E+In+History+of+programming+languages%E2%80%94II%2C+pages+331%E2%80%93367%2E)
9. Luigi Pietro Cordella, Pasquale Foggia, Carlo Sansone, and Mario Vento. 2001. An improved algorithm for matching large graphs. In 3rd IAPR-TC15 workshop on graph-based representations in pattern recognition, pages 149–159. [find](https://scholar.google.com/scholar?q=Luigi+Pietro+Cordella%2C+Pasquale+Foggia%2C+Carlo+Sansone%2C+and+Mario+Vento%2E+2001%2E+An+improved+algorithm+for+matching+large+graphs%2E+In+3rd+IAPR%2DTC15+workshop+on+graph%2Dbased+representations+in+pattern+recog)
10. Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Bhavana+Dalvi%2C+Lifu+Huang%2C+Niket+Tandon%2C+Wen%2Dtau+Yih%2C+and+Peter+Clark%2E+2018%2E+Tracking+state+changes+in+procedural+text%3A+a+challenge+dataset+and+models+for+process+paragraph+comprehension%2E+In+Proceedin)
11. Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. Incoder: A generative model for code infilling and synthesis. arXiv preprint arXiv:2204.05999. [find](https://scholar.google.com/scholar?q=Daniel+Fried%2C+Armen+Aghajanyan%2C+Jessy+Lin%2C+Sida+Wang%2C+Eric+Wallace%2C+Freda+Shi%2C+Ruiqi+Zhong%2C+Wen%2Dtau+Yih%2C+Luke+Zettlemoyer%2C+and+Mike+Lewis%2E+2022%2E+Incoder%3A+A+generative+model+for+code+infilling+and+synt)
12. Emden Gansner, Eleftherios Koutsofios, and Stephen North. 2006. Drawing graphs with dot. [find](https://scholar.google.com/scholar?q=Emden+Gansner%2C+Eleftherios+Koutsofios%2C+and+Stephen+North%2E+2006%2E+Drawing+graphs+with+dot%2E)
13. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. [What Makes Good In-Context Examples for GPT-$3$?](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/) arXiv:2101.06804 [cs]. ArXiv: 2101.06804.
14. Kaixin Ma, Filip Ilievski, Jonathan Francis, Eric Nyberg, and Alessandro Oltramari. 2022. Coalescing global and local information for procedural text understanding. arXiv preprint arXiv:2208.12848. [find](https://scholar.google.com/scholar?q=Kaixin+Ma%2C+Filip+Ilievski%2C+Jonathan+Francis%2C+Eric+Nyberg%2C+and+Alessandro+Oltramari%2E+2022%2E+Coalescing+global+and+local+information+for+procedural+text+understanding%2E+arXiv+preprint+arXiv%3A2208%2E12848%2E)
15. Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yiming Yang, and Eduard Hovy. 2021a. Could you give me a hint ? generating inference graphs for defeasible reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 5138–5147, Online. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Dheeraj+Rajagopal%2C+Niket+Tandon%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2021a%2E+Could+you+give+me+a+hint+%3F+generating+inference+graphs+for+defeasible+reasoning%2E+In+Findings+of+the+Association+for+C)
16. Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, and Eduard Hovy. 2021b. Think about it! improving defeasible reasoning by first modeling the question scenario. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6291–6310. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Niket+Tandon%2C+Dheeraj+Rajagopal%2C+Peter+Clark%2C+Yiming+Yang%2C+and+Eduard+Hovy%2E+2021b%2E+Think+about+it%21+improving+defeasible+reasoning+by+first+modeling+the+question+scenario%2E+In+Proceedings+o)
17. Aman Madaan and Yiming Yang. 2021. Neural language modeling for contextualized temporal graph generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 864–881, Online. Association for Computat… [find](https://scholar.google.com/scholar?q=Aman+Madaan+and+Yiming+Yang%2E+2021%2E+Neural+language+modeling+for+contextualized+temporal+graph+generation%2E+In+Proceedings+of+the+2021+Conference+of+the+North+American+Chapter+of+the+Association+for+Com)
18. Swarnadeep Saha, Prateek Yadav, Lisa Bauer, and Mohit Bansal. 2021. ExplaGraphs: An Explanation Graph Generation Task for Structured Commonsense Reasoning. arXiv:2104.07644 [cs]. ArXiv: 2104.07644. [find](https://scholar.google.com/scholar?q=Swarnadeep+Saha%2C+Prateek+Yadav%2C+Lisa+Bauer%2C+and+Mohit+Bansal%2E+2021%2E+ExplaGraphs%3A+An+Explanation+Graph+Generation+Task+for+Structured+Commonsense+Reasoning%2E+arXiv%3A2104%2E07644+%5Bcs%5D%2E+ArXiv%3A+2104%2E07644%2E)
19. Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras, Niket Tandon, Peter Clark, and Yejin Choi. 2021. proScript: Partially Ordered Scripts Generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2138–2149, Punta Cana, Dominican Republic. Association for Computat… [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Chandra+Bhagavatula%2C+Ronan+Le+Bras%2C+Niket+Tandon%2C+Peter+Clark%2C+and+Yejin+Choi%2E+2021%2E+proScript%3A+Partially+Ordered+Scripts+Generation%2E+In+Findings+of+the+Association+for+Computationa)
20. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696. [find](https://scholar.google.com/scholar?q=Nasrin+Mostafazadeh%2C+Nathanael+Chambers%2C+Xiaodong+He%2C+Devi+Parikh%2C+Dhruv+Batra%2C+Lucy+Vanderwende%2C+Pushmeet+Kohli%2C+and+James+Allen%2E+2016%2E+A+corpus+and+evaluation+framework+for+deeper+understanding+of+c)
21. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2022. [Progprompt: Generating situated robot task plans using large language models](https://md.research.elamur.ai/free/en/papers/singh-2022-progprompt-generating-situated-robot-task/). arXiv preprint arXiv:2209.11302.
22. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. arXiv preprint. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+A+conversational+paradigm+for+program+synthesis%2E+arXiv+preprint%2E)
23. Shao-Hua Sun, Te-Lin Wu, and Joseph J Lim. 2019. Program guided agent. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Shao%2DHua+Sun%2C+Te%2DLin+Wu%2C+and+Joseph+J+Lim%2E+2019%2E+Program+guided+agent%2E+In+International+Conference+on+Learning+Representations%2E)
24. Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2021. Synchromesh: Reliable code generation from pre-trained language models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Gabriel+Poesia%2C+Alex+Polozov%2C+Vu+Le%2C+Ashish+Tiwari%2C+Gustavo+Soares%2C+Christopher+Meek%2C+and+Sumit+Gulwani%2E+2021%2E+Synchromesh%3A+Reliable+code+generation+from+pre%2Dtrained+language+models%2E+In+International+)
25. Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018. Virtualhome: Simulating household activities via programs. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 8494–8502.… [find](https://scholar.google.com/scholar?q=Xavier+Puig%2C+Kevin+Ra%2C+Marko+Boben%2C+Jiaman+Li%2C+Tingwu+Wang%2C+Sanja+Fidler%2C+and+Antonio+Torralba%2E+2018%2E+Virtualhome%3A+Simulating+household+activities+via+programs%2E+In+2018+IEEE+Conference+on+Computer+Vis)
26. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
27. Dheeraj Rajagopal, Aman Madaan, Niket Tandon, Yiming Yang, Shrimai Prabhumoye, Abhilasha Ravichander, Peter Clark, and Eduard Hovy. 2021. Curie: An iterative querying approach for reasoning about situations. [find](https://scholar.google.com/scholar?q=Dheeraj+Rajagopal%2C+Aman+Madaan%2C+Niket+Tandon%2C+Yiming+Yang%2C+Shrimai+Prabhumoye%2C+Abhilasha+Ravichander%2C+Peter+Clark%2C+and+Eduard+Hovy%2E+2021%2E+Curie%3A+An+iterative+querying+approach+for+reasoning+about+situ)
28. Nils Reimers and Iryna Gurevych. 2019. [Sentence-BERT: Sentence embeddings using Siamese BERT-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
29. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021. [Learning To Retrieve Prompts for In-Context Learning](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/). arXiv:2112.08633 [cs]. ArXiv: 2112.08633.
30. Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. 2019. WIQA: A dataset for “what if...” reasoning over procedural text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La… [find](https://scholar.google.com/scholar?q=Niket+Tandon%2C+Bhavana+Dalvi%2C+Keisuke+Sakaguchi%2C+Peter+Clark%2C+and+Antoine+Bosselut%2E+2019%2E+WIQA%3A+A+dataset+for+%E2%80%9Cwhat+if%2E%2E%2E%E2%80%9D+reasoning+over+procedural+text%2E+In+Proceedings+of+the+2019+Conference+on+Empir)
31. Steven L Tanimoto. 1987. The elements of artificial intelligence: an introduction using LISP. Computer Science Press, Inc. [find](https://scholar.google.com/scholar?q=Steven+L+Tanimoto%2E+1987%2E+The+elements+of+artificial+intelligence%3A+an+introduction+using+LISP%2E+Computer+Science+Press%2C+Inc%2E)
32. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). In Advances in Neural Information Processing Systems 32: Annual Conference on Neural I…
33. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859. [find](https://scholar.google.com/scholar?q=Yue+Wang%2C+Weishi+Wang%2C+Shafiq+Joty%2C+and+Steven+CH+Hoi%2E+2021%2E+Codet5%3A+Identifier%2Daware+unified+pre%2Dtrained+encoder%2Ddecoder+models+for+code+understanding+and+generation%2E+arXiv+preprint+arXiv%3A2109%2E00859%2E)
34. Lilian D. A. Wanzare, Alessandra Zarcone, Stefan Thater, and Manfred Pinkal. 2016. A crowdsourced database of event sequence descriptions for the acquisition of high-quality script knowledge. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3… [find](https://scholar.google.com/scholar?q=Lilian+D%2E+A%2E+Wanzare%2C+Alessandra+Zarcone%2C+Stefan+Thater%2C+and+Manfred+Pinkal%2E+2016%2E+A+crowdsourced+database+of+event+sequence+descriptions+for+the+acquisition+of+high%2Dquality+script+knowledge%2E+In+Proce)
35. Yuhuai Wu, Albert Q Jiang, Wenda Li, Markus N Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. 2022. [Autoformalization with large language models](https://md.research.elamur.ai/free/en/papers/wu-2022-autoformalization-large-language-models/). arXiv preprint arXiv:2205.12615.
36. Frank F Xu, Uri Alon, Graham Neubig, and Vincent J Hellendoorn. 2022. [A systematic evaluation of large language models of code](https://md.research.elamur.ai/free/en/papers/xu-2022-systematic-evaluation-large-language-models/). arXiv preprint arXiv:2202.13169.
37. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q%2E+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+Bertscore%3A+Evaluating+text+generation+with+BERT%2E+In+8th+International+Conference+on+Learning+Representations%2C+ICLR+2)
38. Shuyan Zhou, Pengcheng Yin, and Graham Neubig. 2022. Hierarchical control of situated agents through natural language. In Workshop on Structured and Unstructured Knowledge Integration (SUKI), Seattle, USA. [find](https://scholar.google.com/scholar?q=Shuyan+Zhou%2C+Pengcheng+Yin%2C+and+Graham+Neubig%2E+2022%2E+Hierarchical+control+of+situated+agents+through+natural+language%2E+In+Workshop+on+Structured+and+Unstructured+Knowledge+Integration+%28SUKI%29%2C+Seattle%2C)

*Updated 2 November 2026*