---
title: "Code4Struct — paper by Wang et al. (2023) — summary and ideas"
description: "CODE4STRUCT tests whether large language models trained on both code and natural language… 30 references, cited by 66 papers here."
authors: ["Xingyao Wang", "Sha Li", "Heng Ji"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.202/"
doi: "10.18653/v1/2023.acl-long.202"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/wang-2023-code4struct-code-generation-few-shot/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2023-code4struct-code-generation-few-shot/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/) / Code4Struct: Code Generation for Few-Shot Event Structure Prediction

# Code4Struct: Code Generation for Few-Shot Event Structure Prediction

**Authors:** [Xingyao Wang](https://md.research.elamur.ai/free/en/authors/xingyao-wang/), [Sha Li](https://md.research.elamur.ai/free/en/authors/sha-li/) and [Heng Ji](https://md.research.elamur.ai/free/en/authors/heng-ji/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.202/) · [PDF](https://aclanthology.org/2023.acl-long.202.pdf) · [DOI 10.18653/v1/2023.acl-long.202](https://doi.org/10.18653/v1/2023.acl-long.202) · [OpenAlex](https://openalex.org/W4385571184)

**Citations:** 62 citations (OpenAlex, 2026-10-14) · [66](https://md.research.elamur.ai/free/en/papers/wang-2023-code4struct-code-generation-few-shot/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
CODE4STRUCT tests whether large language models trained on both code and natural language can handle structured prediction when the target structure is written as a program. The authors use Event Argument Extraction (EAE) as a case study. Given a sentence with a marked trigger and a known event type, the model must fill each argument role with entity mentions. The ACE entity and event ontology is rewritten as Python classes:

- Event types become classes that inherit from their parent type.
- Argument roles become constructor parameters. Their type annotations restrict which entity types are allowed, and each defaults to an empty list.
- Docstrings carry natural-language descriptions and role templates.

The LLM is prompted to complete a partial instantiation of the event class, optionally after k in-context examples, and the arguments are read from the generated code. With only 20 examples per event type, this matches supervised systems trained on far more data and clearly beats a 20-shot SOTA baseline. The work shows that programming-language features offer an easy way to add ontology knowledge and output constraints to few-shot extraction.

## Key points
- 20-shot CODE4STRUCT performs comparably to fully supervised models trained on 4,202 instances. It surpasses the SOTA trained on the same 20-shot data by 29.5% absolute F1, and even its 0-shot version beats the SOTA at 20 and 50 shots.
- Class inheritance allows zero-resource event types to borrow 10-shot examples from a sibling type. This gives about 12% absolute F1 over the zero-shot baseline.
- With enough in-context examples, code prompts generally outperform two kinds of text-based prompts across several LLMs. Most experiments use code-davinci-002, and text-davinci API runs cost about 700 USD.
- Experiments rely on the ACE05-E data and its two-level event hierarchy.
- The method has several limitations:
  - It assumes gold triggers and a human-curated ontology.
  - Inference is slow and costly compared with much smaller supervised models.
  - In-context examples were not optimized for selection or ordering.
- Code is released at github.com/xingyaoww/code4struct.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Illinois Urbana-Champaign](https://md.research.elamur.ai/free/en/universities/university-of-illinois-urbana-champaign/) — University, Urbana, United States · 108 works · 67,432 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [👁️ Computer Vision and Pattern Recognition](https://md.research.elamur.ai/free/en/shelves/computer-science/computer-vision-and-pattern-recognition/) · 1,282 works

🧭 Topics (1): [💻 Code & programming languages](https://md.research.elamur.ai/free/en/topics/code-and-programming/) · 56 works

## Cite

Cite as: Xingyao Wang, Sha Li and Heng Ji. 2023. Code4Struct: Code Generation for Few-Shot Event Structure Prediction. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{wang2023code4struct,
  title = {{Code4Struct: Code Generation for Few-Shot Event Structure Prediction}},
  author = {Xingyao Wang, Sha Li and Heng Ji},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.202},
  url = {https://doi.org/10.18653/v1/2023.acl-long.202}
}
```

- [← previous Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Data](https://md.research.elamur.ai/free/en/papers/shum-2023-automatic-prompt-augmentation-selection-chain/)
- [next → Deductive Verification of Chain-of-Thought Reasoning](https://md.research.elamur.ai/free/en/papers/ling-2023-deductive-verification-chain-thought-reasoning/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/5/)

## References

30 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. [Abstract Meaning Representation for sembanking](https://md.research.elamur.ai/free/en/papers/banarescu-2013-abstract-meaning-representation-sembanking/). In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discou…
2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). Advances in neural information processing systems, 33:1877–1901.
3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. [Evaluating large language models trained on code](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/). arXiv preprint arXiv:2107.03374.
4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. [Palm: Scaling language modeling with pathways](https://md.research.elamur.ai/free/en/papers/chowdhery-2022-palm-scaling-language-modeling-pathways/). arXiv preprint arXiv:2204.02311.
5. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. [The automatic content extraction (ACE) program – tasks, data, and evaluation](https://md.research.elamur.ai/free/en/papers/doddington-2004-automatic-content-extraction-ace-program/). In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbo…
6. Xinya Du and Claire Cardie. 2020. [Event extraction by answering (almost) natural questions](https://md.research.elamur.ai/free/en/papers/du-2020-event-extraction-answering-almost-natural/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 671–683, Online. Association for Computational Linguistics.
7. Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. 2022. Pal: Program-aided language models. ArXiv, abs/2211.10435. makes good in-context examples for gpt-3? In Workshop on Knowledge Extraction and Integration for Deep Learning Architectures; Dee… [find](https://scholar.google.com/scholar?q=Luyu+Gao%2C+Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Pengfei+Liu%2C+Yiming+Yang%2C+Jamie+Callan%2C+and+Graham+Neubig%2E+2022%2E+Pal%3A+Program%2Daided+language+models%2E+ArXiv%2C+abs%2F2211%2E10435%2E+makes+good+in%2Dcontext+examples)
8. Maria Halkidi, Yannis Batistakis, and Michalis Vazirgiannis. 2001. On clustering validation techniques. Journal of Intelligent Information Systems, 17:107– 145. [find](https://scholar.google.com/scholar?q=Maria+Halkidi%2C+Yannis+Batistakis%2C+and+Michalis+Vazirgiannis%2E+2001%2E+On+clustering+validation+techniques%2E+Journal+of+Intelligent+Information+Systems%2C+17%3A107%E2%80%93+145%2E)
9. Jian Liu, Yubo Chen, Kang Liu, Wei Bi, and Xiaojiang Liu. 2020. [Event extraction as machine reading comprehension](https://md.research.elamur.ai/free/en/papers/liu-2020-event-extraction-as-machine-reading/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1641–1651, Online. Association for Computational Linguistics.
10. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. [Training compute-optimal large language models](https://md.research.elamur.ai/free/en/papers/hoffmann-2022-training-compute-optimal-large-language/). arXiv preprint arXiv:2203.15556.
11. I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022. [DEGREE: A data-efficient generation-based event extraction model](https://md.research.elamur.ai/free/en/papers/hsu-2022-degree-data-efficient-generation-based/). In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: H…
12. Kung-Hsiang Huang, Sam Tang, and Nanyun Peng. 2021. Document-level entity-based extraction as template generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5257–5269, Online and Punta Cana, Dominican Republic. Association for Computational Ling… [find](https://scholar.google.com/scholar?q=Kung%2DHsiang+Huang%2C+Sam+Tang%2C+and+Nanyun+Peng%2E+2021%2E+Document%2Dlevel+entity%2Dbased+extraction+as+template+generation%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Proces)
13. Heng Ji and Ralph Grishman. 2008. Refining event extraction through unsupervised cross-document inference. In In Proceedings of the Annual Meeting of the Association of Computational Linguistics (ACL 2008). Ohio, USA. [find](https://scholar.google.com/scholar?q=Heng+Ji+and+Ralph+Grishman%2E+2008%2E+Refining+event+extraction+through+unsupervised+cross%2Ddocument+inference%2E+In+In+Proceedings+of+the+Annual+Meeting+of+the+Association+of+Computational+Linguistics+%28ACL+)
14. Qi Li, Heng Ji, and Liang Huang. 2013. [Joint event extraction via structured prediction with global features](https://md.research.elamur.ai/free/en/papers/li-2013-joint-event-extraction-structured-prediction/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 73–82, Sofia, Bulgaria. Association for Computational Linguistics.
15. Sha Li, Heng Ji, and Jiawei Han. 2021. [Document-level event argument extraction by conditional generation](https://md.research.elamur.ai/free/en/papers/li-2021-document-level-event-argument-extraction/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 894–908, Online. Association for Computa…
16. Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. 2020. [A joint neural model for information extraction with global features](https://md.research.elamur.ai/free/en/papers/lin-2020-joint-neural-model-information-extraction/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7999–8009, Online. Association for Computational Linguistics.
17. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. [find](https://scholar.google.com/scholar?q=Jiachang+Liu%2C+Dinghan+Shen%2C+Yizhe+Zhang%2C+Bill+Dolan%2C+Lawrence+Carin%2C+and+Weizhu+Chen%2E+2021%2E)
18. What Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. [Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity](https://md.research.elamur.ai/free/en/papers/lu-2022-fantastically-ordered-prompts-where-find/). In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper…
19. Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, Meng Liao, and Shaoyi Chen. 2021. [Text2Event: Controllable sequence-to-structure generation for end-to-end event extraction](https://md.research.elamur.ai/free/en/papers/lu-2021-text2event-controllable-sequence-structure-generation/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the…
20. Aman Madaan, Shuyan Zhou, Uri Alon, Yiming Yang, and Graham Neubig. 2022. Language models of code are few-shot commonsense learners. arXiv preprint arXiv:2210.07128. [find](https://scholar.google.com/scholar?q=Aman+Madaan%2C+Shuyan+Zhou%2C+Uri+Alon%2C+Yiming+Yang%2C+and+Graham+Neubig%2E+2022%2E+Language+models+of+code+are+few%2Dshot+commonsense+learners%2E+arXiv+preprint+arXiv%3A2210%2E07128%2E)
21. L. A. Miller. 1981. Natural language programming: Styles, strategies, and contrasts. IBM Systems Journal, 20(2):184–215. [find](https://scholar.google.com/scholar?q=L%2E+A%2E+Miller%2E+1981%2E+Natural+language+programming%3A+Styles%2C+strategies%2C+and+contrasts%2E+IBM+Systems+Journal%2C+20%282%29%3A184%E2%80%93215%2E)
22. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. [Rethinking the role of demonstrations: What makes in-context learning work?](https://md.research.elamur.ai/free/en/papers/min-2022-rethinking-role-demonstrations-what-makes/) ArXiv, abs/2202.12837.
23. Thien Huu Nguyen, Kyunghyun Cho, and Ralph Grishman. 2016. [Joint event extraction via recurrent neural networks](https://md.research.elamur.ai/free/en/papers/nguyen-2016-joint-event-extraction-recurrent-neural/). In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 300–309, San Diego, California. As…
24. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A conversational paradigm for program synthesis. arXiv preprint. [find](https://scholar.google.com/scholar?q=Erik+Nijkamp%2C+Bo+Pang%2C+Hiroaki+Hayashi%2C+Lifu+Tu%2C+Huan+Wang%2C+Yingbo+Zhou%2C+Silvio+Savarese%2C+and+Caiming+Xiong%2E+2022%2E+A+conversational+paradigm+for+program+synthesis%2E+arXiv+preprint%2E)
25. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
26. Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. arXiv preprint arXiv:2101.05779. [find](https://scholar.google.com/scholar?q=Giovanni+Paolini%2C+Ben+Athiwaratkun%2C+Jason+Krone%2C+Jie+Ma%2C+Alessandro+Achille%2C+Rishita+Anubhai%2C+Cicero+Nogueira+dos+Santos%2C+Bing+Xiang%2C+and+Stefano+Soatto%2E+2021%2E+Structured+prediction+as+translation+bet)
27. Marc M. Sebrechts and Paul Gross. 1985. Programming in natural language: A descriptive analysis. Behavior Research Methods, Instruments, & Computers, 17:268–274. [find](https://scholar.google.com/scholar?q=Marc+M%2E+Sebrechts+and+Paul+Gross%2E+1985%2E+Programming+in+natural+language%3A+A+descriptive+analysis%2E+Behavior+Research+Methods%2C+Instruments%2C+%26+Computers%2C+17%3A268%E2%80%93274%2E)
28. Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg. 2022. Prog-Prompt: Generating situated robot task plans using large language models. [find](https://scholar.google.com/scholar?q=Ishika+Singh%2C+Valts+Blukis%2C+Arsalan+Mousavian%2C+Ankit+Goyal%2C+Danfei+Xu%2C+Jonathan+Tremblay%2C+Dieter+Fox%2C+Jesse+Thomason%2C+and+Animesh+Garg%2E+2022%2E+Prog%2DPrompt%3A+Generating+situated+robot+task+plans+using+la)
29. Shao-Hua Sun, Te-Lin Wu, and Joseph J Lim. 2019. Program guided agent. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Shao%2DHua+Sun%2C+Te%2DLin+Wu%2C+and+Joseph+J+Lim%2E+2019%2E+Program+guided+agent%2E+In+International+Conference+on+Learning+Representations%2E)
30. David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. 2019. [Entity, relation, and event extraction with contextualized span representations](https://md.research.elamur.ai/free/en/papers/wadden-2019-entity-relation-event-extraction-contextualized/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua…

*Updated 24 October 2026*