---
title: "Diverse Demonstrations Improve In-context — Levy 2023"
description: "Few-shot prompting (in-context learning) of large LMs works well for semantic parsing when test and training data come from the same distribution."
authors: ["Itay Levy", "Ben Bogin", "Jonathan Berant"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)"
source: "https://aclanthology.org/2023.acl-long.78/"
doi: "10.18653/v1/2023.acl-long.78"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/levy-2023-diverse-demonstrations-improve-context-compositional/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/levy-2023-diverse-demonstrations-improve-context-compositional/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/) / Diverse Demonstrations Improve In-context Compositional Generalization

# Diverse Demonstrations Improve In-context Compositional Generalization

**Authors:** Itay Levy, [Ben Bogin](https://md.research.elamur.ai/free/en/authors/ben-bogin/), [Jonathan Berant](https://md.research.elamur.ai/free/en/authors/jonathan-berant/) · **Year:** 2023 · **Venue:** Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-long.78/) · [PDF](https://aclanthology.org/2023.acl-long.78.pdf) · [DOI 10.18653/v1/2023.acl-long.78](https://doi.org/10.18653/v1/2023.acl-long.78) · [OpenAlex](https://openalex.org/W4385571445)

**Citations:** 61 citations (openalex-snapshot · 2026-10-14) · [78](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Few-shot prompting (in-context learning) of large LMs works well for semantic parsing when test and training data come from the same distribution. It is much harder on compositional splits, where test programs combine familiar pieces into structures never seen in training. The usual approach is to retrieve the training examples most similar to the input. On these splits, those examples tend to resemble each other and miss parts of the target program.

The authors therefore choose demonstrations as a diverse set. Their main method, Cover-LS, works in two steps:
- A finetuned T5 with beam search predicts which local structures (sub-trees of the program's abstract syntax tree) the output probably needs.
- Training examples are then chosen greedily, largest structures first, until the predicted structures are jointly covered.

A simpler variant ignores the input and picks mutually dissimilar examples. The methods were tested on three compositional datasets in two setups: prompting Codex with no finetuning, and finetuning with demonstrations in the input. In the finetuning setup, noisy demonstrations during training reduce over-reliance on them. Diverse selection beat similarity-only retrieval by up to 23 points, showing that a prompt should be judged as a whole set.

## Key points
- Two kinds of diversity are compared. Coverage-based selection (Cover-LS) depends on the input, while dissimilarity-based selection (e.g., DPP) is input-independent. Coverage-based selection performed better.
- The best reported gain is on SMCalFlow-CS, rising from 50.3 to 73.5 accuracy over a Top-K similarity baseline. The method reached state-of-the-art results in several compositional setups.
- Retrievers tested were BM25-Utterance, BM25-Predicted, SBERT and random. Cover-LS beat Top-K with every retriever and varied less across them, with BM25-Utterance performing best.
- Diverse sets need fewer demonstrations to generalize, improve accuracy on hard examples, and contain more unique local structures than Top-K sets.
- Evaluation covers only English data. Code is available at github.com/itayle/diverse-demonstrations.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

78 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 76 not yet.

- [Faith and Fate: Limits of Transformers on Compositionality](https://md.research.elamur.ai/free/en/papers/dziri-2023-faith-fate-limits-transformers-compositionality/)
- [A Survey of Large Language Models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{levy2023diverse,
  title = {{Diverse Demonstrations Improve In-context Compositional Generalization}},
  author = {Levy, Itay and Bogin, Ben and Berant, Jonathan},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.78},
  url = {https://doi.org/10.18653/v1/2023.acl-long.78}
}
```

- [← previous Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes](https://md.research.elamur.ai/free/en/papers/hsieh-2023-distilling-step-step-outperforming-larger/)
- [next → Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://md.research.elamur.ai/free/en/papers/ahia-2023-do-all-languages-cost-same/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/3/)

## References

48 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Chris… [find](https://scholar.google.com/scholar?q=Jacob+Andreas%2C+John+Bufe%2C+David+Burkett%2C+Charles+Chen%2C+Josh+Clausman%2C+Jean+Crawford%2C+Kate+Crim%2C+Jordan+DeLoach%2C+Leah+Dorner%2C+Jason+Eisner%2C+Hao+Fang%2C+Alan+Guo%2C+David+Hall%2C+Kristin+Hayes%2C+Kellie+Hill%2C+D) [our card](https://md.research.elamur.ai/free/en/papers/andreas-2020-task-oriented-dialogue-as-dataflow/)
2. Ben Bogin, Shivanshu Gupta, and Jonathan Berant. 2022. Unobserved local structures make compositional generalization hard. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2731–2747, Abu Dhabi, United Arab Emirates. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Ben+Bogin%2C+Shivanshu+Gupta%2C+and+Jonathan+Berant%2E+2022%2E+Unobserved+local+structures+make+compositional+generalization+hard%2E+In+Proceedings+of+the+2022+Conference+on+Empirical+Methods+in+Natural+Languag)
3. Dorian Brown. 2020. Rank-BM25: A Collection of BM25 Algorithms in Python. [find](https://scholar.google.com/scholar?q=Dorian+Brown%2E+2020%2E+Rank%2DBM25%3A+A+Collection+of+BM25+Algorithms+in+Python%2E)
4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… [find](https://scholar.google.com/scholar?q=Tom+B%2E+Brown%2C+Benjamin+Mann%2C+Nick+Ryder%2C+Melanie+Subbiah%2C+Jared+Kaplan%2C+Prafulla+Dhariwal%2C+Arvind+Neelakantan%2C+Pranav+Shyam%2C+Girish+Sastry%2C+Amanda+Askell%2C+Sandhini+Agarwal%2C+Ariel+Herbert%2DVoss%2C+Gretche)
5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde, Jared Kaplan, Harrison Edwards, Yura Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Aleth… [find](https://scholar.google.com/scholar?q=Mark+Chen%2C+Jerry+Tworek%2C+Heewoo+Jun%2C+Qiming+Yuan%2C+Henrique+Ponde%2C+Jared+Kaplan%2C+Harrison+Edwards%2C+Yura+Burda%2C+Nicholas+Joseph%2C+Greg+Brockman%2C+Alex+Ray%2C+Raul+Puri%2C+Gretchen+Krueger%2C+Michael+Petrov%2C+Hei) [our card](https://md.research.elamur.ai/free/en/papers/chen-2021-evaluating-large-language-models-trained/)
6. Yanda Chen, Ruiqi Zhong, Sheng Zha, George Karypis, and He He. 2022. Meta-learning via language model in-context tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 719–730, Dublin, Ireland. Association for Computational L… [find](https://scholar.google.com/scholar?q=Yanda+Chen%2C+Ruiqi+Zhong%2C+Sheng+Zha%2C+George+Karypis%2C+and+He+He%2E+2022%2E+Meta%2Dlearning+via+language+model+in%2Dcontext+tuning%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+)
7. Henry Conklin, Bailin Wang, Kenny Smith, and Ivan Titov. 2021. Meta-learning to compositionally generalize. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p… [find](https://scholar.google.com/scholar?q=Henry+Conklin%2C+Bailin+Wang%2C+Kenny+Smith%2C+and+Ivan+Titov%2E+2021%2E+Meta%2Dlearning+to+compositionally+generalize%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+a)
8. Andrew Drozdov, Nathanael Scharli, Ekin Akyuurek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. ArXiv preprint, abs/2209.15003. [find](https://scholar.google.com/scholar?q=Andrew+Drozdov%2C+Nathanael+Scharli%2C+Ekin+Akyuurek%2C+Nathan+Scales%2C+Xinying+Song%2C+Xinyun+Chen%2C+Olivier+Bousquet%2C+and+Denny+Zhou%2E+2022%2E+Compositional+semantic+parsing+with+large+language+models%2E+ArXiv+pre)
9. Catherine Finegan-Dollak, Jonathan K. Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-SQL evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… [find](https://scholar.google.com/scholar?q=Catherine+Finegan%2DDollak%2C+Jonathan+K%2E+Kummerfeld%2C+Li+Zhang%2C+Karthik+Ramanathan%2C+Sesh+Sadasivam%2C+Rui+Zhang%2C+and+Dragomir+Radev%2E+2018%2E+Improving+text%2Dto%2DSQL+evaluation+methodology%2E+In+Proceedings+of+the) [our card](https://md.research.elamur.ai/free/en/papers/dollak-2018-improving-text-sql-evaluation-methodology/)
10. Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research… [find](https://scholar.google.com/scholar?q=Chelsea+Finn%2C+Pieter+Abbeel%2C+and+Sergey+Levine%2E+2017%2E+Model%2Dagnostic+meta%2Dlearning+for+fast+adaptation+of+deep+networks%2E+In+Proceedings+of+the+34th+International+Conference+on+Machine+Learning%2C+ICML+2)
11. Daniel Furrer, Marc van Zee, Nathan Scales, and Nathanael Scharli. 2020. Compositional generalization in semantic parsing: Pre-training vs. specialized architectures. ArXiv preprint, abs/2007.08970. [find](https://scholar.google.com/scholar?q=Daniel+Furrer%2C+Marc+van+Zee%2C+Nathan+Scales%2C+and+Nathanael+Scharli%2E+2020%2E+Compositional+generalization+in+semantic+parsing%3A+Pre%2Dtraining+vs%2E+specialized+architectures%2E+ArXiv+preprint%2C+abs%2F2007%2E08970%2E)
12. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1–6, Melbourne,… [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+Peters%2C+Michael+Schmitz%2C+and+Luke+Zettlemoyer%2E+2018%2E+AllenNLP%3A+A+deep+semantic+natural+language+processing) [our card](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
13. Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Structurally diverse sampling for sample-efficient training and comprehensive evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4966–4979, Abu Dhabi, United Arab Emirates. Association for Computational L… [find](https://scholar.google.com/scholar?q=Shivanshu+Gupta%2C+Sameer+Singh%2C+and+Matt+Gardner%2E+2022%2E+Structurally+diverse+sampling+for+sample%2Defficient+training+and+comprehensive+evaluation%2E+In+Findings+of+the+Association+for+Computational+Lingui)
14. Jonathan Herzig and Jonathan Berant. 2019. Don’t paraphrase, detect! rapid and effective data collection for semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJ… [find](https://scholar.google.com/scholar?q=Jonathan+Herzig+and+Jonathan+Berant%2E+2019%2E+Don%E2%80%99t+paraphrase%2C+detect%21+rapid+and+effective+data+collection+for+semantic+parsing%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Lan)
15. Arian Hosseini, Ankit Vani, Dzmitry Bahdanau, Alessandro Sordoni, and Aaron Courville. 2022. On the compositional generalization gap of in-context learning. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 272–280, Abu Dhabi, United Arab E… [find](https://scholar.google.com/scholar?q=Arian+Hosseini%2C+Ankit+Vani%2C+Dzmitry+Bahdanau%2C+Alessandro+Sordoni%2C+and+Aaron+Courville%2E+2022%2E+On+the+compositional+generalization+gap+of+in%2Dcontext+learning%2E+In+Proceedings+of+the+Fifth+BlackboxNLP+Wor)
16. Dieuwke Hupkes, Mario Giulianelli, Verna Dankers, Mikel Artetxe, Yanai Elazar, Tiago Pimentel, Christos Christodoulopoulos, Karim Lasri, Naomi Saphra, Arabella Sinclair, Dennis Ulmer, Florian Schottmann, Khuyagbaatar Batsuren, Kaiser Sun, Koustuv Sinha, Leila Khalatbari, Maria Ryskina, Rita Frieske,… [find](https://scholar.google.com/scholar?q=Dieuwke+Hupkes%2C+Mario+Giulianelli%2C+Verna+Dankers%2C+Mikel+Artetxe%2C+Yanai+Elazar%2C+Tiago+Pimentel%2C+Christos+Christodoulopoulos%2C+Karim+Lasri%2C+Naomi+Saphra%2C+Arabella+Sinclair%2C+Dennis+Ulmer%2C+Florian+Schottma)
17. Vishal Kaushal, Ganesh Ramakrishnan, and Rishabh K. Iyer. 2022. Submodlib: A submodular optimization library. ArXiv preprint, abs/2202.10680. — [arXiv:2202.10680](https://arxiv.org/abs/2202.10680) [find](https://scholar.google.com/scholar?q=Vishal+Kaushal%2C+Ganesh+Ramakrishnan%2C+and+Rishabh+K%2E+Iyer%2E+2022%2E+Submodlib%3A+A+submodular+optimization+library%2E+ArXiv+preprint%2C+abs%2F2202%2E10680%2E+%E2%80%94+%5BarXiv%3A2202%2E10680%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2202%2E10680%29)
18. Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020. Measuring compositional generalization: A comprehensive method on… [find](https://scholar.google.com/scholar?q=Daniel+Keysers%2C+Nathanael+Sch%C3%A4rli%2C+Nathan+Scales%2C+Hylke+Buisman%2C+Daniel+Furrer%2C+Sergii+Kashubin%2C+Nikola+Momchev%2C+Danila+Sinopalnikov%2C+Lukasz+Stafiniak%2C+Tibor+Tihon%2C+Dmitry+Tsarkov%2C+Xiao+Wang%2C+Marc+van)
19. Alex Kulesza and Ben Taskar. 2012. Determinantal point processes for machine learning. Foundations and Trends® in Machine Learning, 5(2–3):123–286. [find](https://scholar.google.com/scholar?q=Alex+Kulesza+and+Ben+Taskar%2E+2012%2E+Determinantal+point+processes+for+machine+learning%2E+Foundations+and+Trends%C2%AE+in+Machine+Learning%2C+5%282%E2%80%933%29%3A123%E2%80%93286%2E)
20. Brenden M. Lake. 2019. Compositional generalization through meta sequence-to-sequence learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 9788–9798. [find](https://scholar.google.com/scholar?q=Brenden+M%2E+Lake%2E+2019%2E+Compositional+generalization+through+meta+sequence%2Dto%2Dsequence+learning%2E+In+Advances+in+Neural+Information+Processing+Systems+32%3A+Annual+Conference+on+Neural+Information+Process)
21. Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 o… [find](https://scholar.google.com/scholar?q=Brenden+M%2E+Lake+and+Marco+Baroni%2E+2018%2E+Generalization+without+systematicity%3A+On+the+compositional+skills+of+sequence%2Dto%2Dsequence+recurrent+networks%2E+In+Proceedings+of+the+35th+International+Conferenc)
22. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100–114… [find](https://scholar.google.com/scholar?q=Jiachang+Liu%2C+Dinghan+Shen%2C+Yizhe+Zhang%2C+Bill+Dolan%2C+Lawrence+Carin%2C+and+Weizhu+Chen%2E+2022%2E+What+makes+good+in%2Dcontext+examples+for+GPT%2D3%3F+In+Proceedings+of+Deep+Learning+Inside+Out+%28DeeLIO+2022%29%3A+The) [our card](https://md.research.elamur.ai/free/en/papers/liu-2022-what-makes-good-context-examples/)
23. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. [find](https://scholar.google.com/scholar?q=Ilya+Loshchilov+and+Frank+Hutter%2E+2019%2E+Decoupled+weight+decay+regularization%2E+In+7th+International+Conference+on+Learning+Representations%2C+ICLR+2019%2C+New+Orleans%2C+LA%2C+USA%2C+May+6%2D9%2C+2019%2E+OpenReview%2En)
24. Clara Meister, Martina Forster, and Ryan Cotterell. 2021. Determinantal beam search. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6551–6562, Online.… [find](https://scholar.google.com/scholar?q=Clara+Meister%2C+Martina+Forster%2C+and+Ryan+Cotterell%2E+2021%2E+Determinantal+beam+search%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+and+the+11th+Internation)
25. Joram Meron. 2022. Simplifying semantic annotations of SMCalFlow. In Proceedings of the 18th Joint ACL - ISO Workshop on Interoperable Semantic Annotation within LREC2022, pages 81–85, Marseille, France. European Language Resources Association. [find](https://scholar.google.com/scholar?q=Joram+Meron%2E+2022%2E+Simplifying+semantic+annotations+of+SMCalFlow%2E+In+Proceedings+of+the+18th+Joint+ACL+%2D+ISO+Workshop+on+Interoperable+Semantic+Annotation+within+LREC2022%2C+pages+81%E2%80%9385%2C+Marseille%2C+Fran)
26. Sewon Min, Kenton Lee, Ming-Wei Chang, Kristina Toutanova, and Hannaneh Hajishirzi. 2021. Joint passage ranking for diverse multi-answer retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6997–7008, Online and Punta Cana, Dominican Republic. A… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Kenton+Lee%2C+Ming%2DWei+Chang%2C+Kristina+Toutanova%2C+and+Hannaneh+Hajishirzi%2E+2021%2E+Joint+passage+ranking+for+diverse+multi%2Danswer+retrieval%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+M)
27. Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2791–2809, Seattle, United States. A… [find](https://scholar.google.com/scholar?q=Sewon+Min%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Hannaneh+Hajishirzi%2E+2022%2E+MetaICL%3A+Learning+to+learn+in+context%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+fo)
28. Poojitha Nandigam, Nikhil Rayaprolu, and Manish Shrivastava. 2022. Diverse multi-answer retrieval with determinantal point processes. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2220–2225, Gyeongju, Republic of Korea. International Committee on Computation… [find](https://scholar.google.com/scholar?q=Poojitha+Nandigam%2C+Nikhil+Rayaprolu%2C+and+Manish+Shrivastava%2E+2022%2E+Diverse+multi%2Danswer+retrieval+with+determinantal+point+processes%2E+In+Proceedings+of+the+29th+International+Conference+on+Computation)
29. Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, and Mirella Lapata. 2022. A well-composed text is half done! composition sampling for diverse conditional generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vo… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Gon%C3%A7alo+Sim%C3%B5es%2C+Yao+Zhao%2C+Joshua+Maynez%2C+Dipanjan+Das%2C+Michael+Collins%2C+and+Mirella+Lapata%2E+2022%2E+A+well%2Dcomposed+text+is+half+done%21+composition+sampling+for+diverse+conditional+genera)
30. Inbar Oren, Jonathan Herzig, and Jonathan Berant. 2021. Finding needles in a haystack: Sampling structurally-diverse training sets from synthetic data for compositional generalization. In Proceedings of the 2021 [find](https://scholar.google.com/scholar?q=Inbar+Oren%2C+Jonathan+Herzig%2C+and+Jonathan+Berant%2E+2021%2E+Finding+needles+in+a+haystack%3A+Sampling+structurally%2Ddiverse+training+sets+from+synthetic+data+for+compositional+generalization%2E+In+Proceedings+)
31. Conference on Empirical Methods in Natural Language Processing, pages 10793–10809, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+10793%E2%80%9310809%2C+Online+and+Punta+Cana%2C+Dominican+Republic%2E+Association+for+Computational+Linguistics%2E)
32. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… [find](https://scholar.google.com/scholar?q=Long+Ouyang%2C+Jeff+Wu%2C+Xu+Jiang%2C+Diogo+Almeida%2C+Carroll+L%2E+Wainwright%2C+Pamela+Mishkin%2C+Chong+Zhang%2C+Sandhini+Agarwal%2C+Katarina+Slama%2C+Alex+Ray%2C+John+Schulman%2C+Jacob+Hilton%2C+Fraser+Kelton%2C+Luke+E%2E+Mille)
33. Panupong Pasupat, Yuan Zhang, and Kelvin Guu. 2021. Controllable semantic parsing via retrieval augmentation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7683–7698, Online and Punta Cana, Dominican Republic. Association for Computational Linguisti… [find](https://scholar.google.com/scholar?q=Panupong+Pasupat%2C+Yuan+Zhang%2C+and+Kelvin+Guu%2E+2021%2E+Controllable+semantic+parsing+via+retrieval+augmentation%2E+In+Proceedings+of+the+2021+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C)
34. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
35. Linlu Qiu, Peter Shaw, Panupong Pasupat, Pawel Nowak, Tal Linzen, Fei Sha, and Kristina Toutanova. 2022a. Improving compositional generalization with latent structure and data augmentation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Linlu+Qiu%2C+Peter+Shaw%2C+Panupong+Pasupat%2C+Pawel+Nowak%2C+Tal+Linzen%2C+Fei+Sha%2C+and+Kristina+Toutanova%2E+2022a%2E+Improving+compositional+generalization+with+latent+structure+and+data+augmentation%2E+In+Proceed)
36. Linlu Qiu, Peter Shaw, Panupong Pasupat, Tianze Shi, Jonathan Herzig, Emily Pitler, Fei Sha, and Kristina Toutanova. 2022b. Evaluating the impact of model scale for compositional generalization in semantic parsing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Proces… [find](https://scholar.google.com/scholar?q=Linlu+Qiu%2C+Peter+Shaw%2C+Panupong+Pasupat%2C+Tianze+Shi%2C+Jonathan+Herzig%2C+Emily+Pitler%2C+Fei+Sha%2C+and+Kristina+Toutanova%2E+2022b%2E+Evaluating+the+impact+of+model+scale+for+compositional+generalization+in+sem)
37. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67. [find](https://scholar.google.com/scholar?q=Colin+Raffel%2C+Noam+Shazeer%2C+Adam+Roberts%2C+Katherine+Lee%2C+Sharan+Narang%2C+Michael+Matena%2C+Yanqi+Zhou%2C+Wei+Li%2C+and+Peter+J%2E+Liu%2E+2020%2E+Exploring+the+limits+of+transfer+learning+with+a+unified+text%2Dto%2Dtex)
38. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2019%2E+Sentence%2DBERT%3A+Sentence+embeddings+using+Siamese+BERT%2Dnetworks%2E+In+Proceedings+of+the+2019+Conference+on+Empirical+Methods+in+Natural+Language+Processing+and+the)
39. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389. [find](https://scholar.google.com/scholar?q=Stephen+Robertson+and+Hugo+Zaragoza%2E+2009%2E+The+probabilistic+relevance+framework%3A+Bm25+and+beyond%2E+Foundations+and+Trends%C2%AE+in+Information+Retrieval%2C+3%284%29%3A333%E2%80%93389%2E)
40. Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2655–2671, Seattle, United States. As… [find](https://scholar.google.com/scholar?q=Ohad+Rubin%2C+Jonathan+Herzig%2C+and+Jonathan+Berant%2E+2022%2E+Learning+to+retrieve+prompts+for+in%2Dcontext+learning%2E+In+Proceedings+of+the+2022+Conference+of+the+North+American+Chapter+of+the+Association+for) [our card](https://md.research.elamur.ai/free/en/papers/rubin-2021-learning-retrieve-prompts-context-learning/)
41. Peter Shaw, Ming-Wei Chang, Panupong Pasupat, and Kristina Toutanova. 2021. Compositional generalization and natural language variation: Can a semantic parsing approach handle both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International… [find](https://scholar.google.com/scholar?q=Peter+Shaw%2C+Ming%2DWei+Chang%2C+Panupong+Pasupat%2C+and+Kristina+Toutanova%2E+2021%2E+Compositional+generalization+and+natural+language+variation%3A+Can+a+semantic+parsing+approach+handle+both%3F+In+Proceedings+of+)
42. Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang, Mari Ostendorf, Luke Zettlemoyer, Noah A. Smith, and Tao Yu. 2022. Selective annotation makes language models better few-shot learners. ArXiv preprint, abs/2209.01975. [find](https://scholar.google.com/scholar?q=Hongjin+Su%2C+Jungo+Kasai%2C+Chen+Henry+Wu%2C+Weijia+Shi%2C+Tianlu+Wang%2C+Jiayi+Xin%2C+Rui+Zhang%2C+Mari+Ostendorf%2C+Luke+Zettlemoyer%2C+Noah+A%2E+Smith%2C+and+Tao+Yu%2E+2022%2E+Selective+annotation+makes+language+models+bet)
43. Lappoon R. Tang and Raymond J. Mooney. 2001. Using multiple clause constructors in inductive logic programming for semantic parsing. In ECML. [find](https://scholar.google.com/scholar?q=Lappoon+R%2E+Tang+and+Raymond+J%2E+Mooney%2E+2001%2E+Using+multiple+clause+constructors+in+inductive+logic+programming+for+semantic+parsing%2E+In+ECML%2E)
44. Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. 2022. Training data is more valuable than you think: A simple and effective method by retrieving from training data. In Proceedings of the 60th Annual Meeting of the Association for Computational… [find](https://scholar.google.com/scholar?q=Shuohang+Wang%2C+Yichong+Xu%2C+Yuwei+Fang%2C+Yang+Liu%2C+Siqi+Sun%2C+Ruochen+Xu%2C+Chenguang+Zhu%2C+and+Michael+Zeng%2E+2022%2E+Training+data+is+more+valuable+than+you+think%3A+A+simple+and+effective+method+by+retrieving)
45. Xi Ye, Srini Iyer, Asli Celikyilmaz, Ves Stoyanov, Greg Durrett, and Ramakanth Pasunuru. 2022. Complementary explanations for effective in-context learning. ArXiv preprint, abs/2211.13892. [find](https://scholar.google.com/scholar?q=Xi+Ye%2C+Srini+Iyer%2C+Asli+Celikyilmaz%2C+Ves+Stoyanov%2C+Greg+Durrett%2C+and+Ramakanth+Pasunuru%2E+2022%2E+Complementary+explanations+for+effective+in%2Dcontext+learning%2E+ArXiv+preprint%2C+abs%2F2211%2E13892%2E)
46. Pengcheng Yin, Hao Fang, Graham Neubig, Adam Pauls, Emmanouil Antonios Platanios, Yu Su, Sam Thomson, and Jacob Andreas. 2021. Compositional generalization for neural semantic parsing via span-level supervised attention. In Proceedings of the 2021 Conference of the North American Chapter of the Asso… [find](https://scholar.google.com/scholar?q=Pengcheng+Yin%2C+Hao+Fang%2C+Graham+Neubig%2C+Adam+Pauls%2C+Emmanouil+Antonios+Platanios%2C+Yu+Su%2C+Sam+Thomson%2C+and+Jacob+Andreas%2E+2021%2E+Compositional+generalization+for+neural+semantic+parsing+via+span%2Dlevel+s)
47. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, Vol. 2. [find](https://scholar.google.com/scholar?q=John+M%2E+Zelle+and+Raymond+J%2E+Mooney%2E+1996%2E+Learning+to+parse+database+queries+using+inductive+logic+programming%2E+In+AAAI%2FIAAI%2C+Vol%2E+2%2E)
48. Yury Zemlyanskiy, Michiel de Jong, Joshua Ainslie, Panupong Pasupat, Peter Shaw, Linlu Qiu, Sumit Sanghai, and Fei Sha. 2022. Generate-and-retrieve: Use your predictions to improve retrieval for semantic parsing. In Proceedings of the 29th International Conference on Computational Linguistics, pages… [find](https://scholar.google.com/scholar?q=Yury+Zemlyanskiy%2C+Michiel+de+Jong%2C+Joshua+Ainslie%2C+Panupong+Pasupat%2C+Peter+Shaw%2C+Linlu+Qiu%2C+Sumit+Sanghai%2C+and+Fei+Sha%2E+2022%2E+Generate%2Dand%2Dretrieve%3A+Use+your+predictions+to+improve+retrieval+for+seman)