---
title: "Few-Shot Text Generation with Natural Language Instructions"
description: "Schick and Schütze introduce GENPET, an extension of pattern-exploiting training for text-to-text tasks. 46 references, cited by 63 papers here."
authors: ["Timo Schick", "Hinrich Schütze"]
year: 2021
venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/2021.emnlp-main.32/"
doi: "10.18653/v1/2021.emnlp-main.32"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schick-2021-few-shot-text-generation-natural/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schick-2021-few-shot-text-generation-natural/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Few-Shot Text Generation with Natural Language Instructions

# Few-Shot Text Generation with Natural Language Instructions

**Authors:** [Timo Schick](https://md.research.elamur.ai/free/en/authors/timo-schick/) and [Hinrich Schütze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.32/) · [PDF](https://aclanthology.org/2021.emnlp-main.32.pdf) · [DOI 10.18653/v1/2021.emnlp-main.32](https://doi.org/10.18653/v1/2021.emnlp-main.32) · [OpenAlex](https://openalex.org/W3212893438)

**Citations:** 103 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/schick-2021-few-shot-text-generation-natural/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Natural-language task descriptions help pretrained language models in few-shot classification, but earlier methods such as PET could not handle open-ended generation. Schick and Schütze introduce GENPET, an extension of pattern-exploiting training for text-to-text tasks. It fine-tunes PEGASUS-large, a 568M-parameter encoder-decoder model, using both textual instructions and a small set of labelled examples. The method addresses three problems:
- **Using the instruction well:** part of the instruction is fed to the decoder as a "decoder prefix" rather than placed entirely in the encoder input.
- **Sensitivity to prompt wording:** several pattern/prefix pairs are combined through knowledge distillation, with instructions sampled probabilistically.
- **Overfitting:** the method relies on unsupervised scoring and joint training.

The authors test GENPET on six English summarization and headline generation datasets, with 10 or 100 training examples and in a zero-shot setting. In the few-shot settings, GENPET consistently beats standard fine-tuning and a baseline that adds only a mask token. The work shows that instruction-based few-shot learning can extend beyond classification to generation.

## Key points
- **Backbone model:** all experiments use PEGASUS-large, a 16-layer Transformer pretrained with gap-sentence generation. It was chosen because it is the only PEGASUS variant publicly released without downstream fine-tuning.
- **Results with 10 examples:** across the three datasets in the ablation table (including AESLC and XSum), GENPET reaches ROUGE-1 of about 27–36. Plain PEGASUS scores roughly 9–30 and PEGASUS-M (initial mask only) roughly 17–34.
- **Ablation findings:**
  - Merging all instructions works better than using only the single best one.
  - Moving the instruction into the encoder ("no dec. prefix") sharply hurts one task, where ROUGE-1 falls from 27.19 to 15.49.
  - Removing joint training lowers scores on all three tasks.
- **Zero-shot behaviour:** in a Gigaword example, plain PEGASUS tends to copy input text word for word and PEGASUS-M invents facts. GENPET produces a fluent, factual headline. GENPET also leads on average in zero-shot, though not on every task.
- **Code availability:** code and few-shot data splits are at github.com/timoschick/pet.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 44 works · 55,193 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [✍️ Text generation](https://md.research.elamur.ai/free/en/topics/text-generation/) · 145 works

## Cite

Cite as: Timo Schick and Hinrich Schütze. 2021. Few-Shot Text Generation with Natural Language Instructions. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schick2021few,
  title = {{Few-Shot Text Generation with Natural Language Instructions}},
  author = {Timo Schick and Hinrich Schütze},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.32},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.32}
}
```

- [← previous Paragraph-level Rationale Extraction through Regularization: A case study on European Court of Human Rights Cases](https://md.research.elamur.ai/free/en/papers/chalkidis-2021-paragraph-level-rationale-extraction-through/)
- [next → On Generative Spoken Language Modeling from Raw Audio](https://md.research.elamur.ai/free/en/papers/lakhotia-2021-generative-spoken-language-modeling-raw/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

46 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
2. Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147– 2157, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Jiaao+Chen%2C+Zichao+Yang%2C+and+Diyi+Yang%2E+2020%2E+Mix%2DText%3A+Linguistically%2Dinformed+interpolation+of+hidden+space+for+semi%2Dsupervised+text+classification%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+)
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
4. Zi-Yi Dou, Keyi Yu, and Antonios Anastasopoulos. 2019. [Investigating meta-learning algorithms for low-resource natural language understanding tasks](https://md.research.elamur.ai/free/en/papers/dou-2019-investigating-meta-learning-algorithms-low/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag…
5. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. Computing Research Repository, arXiv:2102.01017. [find](https://scholar.google.com/scholar?q=Yanai+Elazar%2C+Nora+Kassner%2C+Shauli+Ravfogel%2C+Abhilasha+Ravichander%2C+Eduard+Hovy%2C+Hinrich+Sch%C3%BCtze%2C+and+Yoav+Goldberg%2E+2021%2E+Measuring+and+improving+consistency+in+pretrained+language+models%2E+Computing+)
6. Jiatao Gu, Yong Wang, Yun Chen, Victor O. K. Li, and Kyunghyun Cho. 2018. [Meta-learning for low-resource neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2018-meta-learning-low-resource-neural/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3622–3631, Brussels, Belgium. Association for Computational Linguistic…
7. Junxian He, Wojciech Kryściński, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2020. [CTRLsum: Towards generic controllable text summarization](https://md.research.elamur.ai/free/en/papers/he-2022-ctrlsum-generic-controllable-text-summarization/). Computing Research Repository, arXiv:2012.04281.
8. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. [Teaching machines to read and comprehend](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/). In Advances in Neural Information Processing Systems, volume 28, pages 1693–1701. Curran Associates, Inc.
9. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). Computing Research Repository, arXiv:1503.02531.
10. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
11. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Transactions of the Association for Computational Linguistics, 8:423–438.
12. Nora Kassner and Hinrich Schütze. 2020. [Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly](https://md.research.elamur.ai/free/en/papers/kassner-2020-negated-misprimed-probes-pretrained-language/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811–7818, Online. Association for Computational Linguistics.
13. Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. [CTRL: A conditional transformer language model for controllable generation](https://md.research.elamur.ai/free/en/papers/keskar-2019-ctrl-conditional-transformer-language-model/). Computing Research Repository, arXiv:1909.05858.
14. Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. [Abstractive summarization of Reddit posts with multi-level memory networks](https://md.research.elamur.ai/free/en/papers/kim-2019-abstractive-summarization-reddit-posts-multi/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short…
15. Rebecca Knowles and Philipp Koehn. 2016. Neural interactive translation prediction. In Proceedings of the Association for Machine Translation in the Americas, pages 107–120. [find](https://scholar.google.com/scholar?q=Rebecca+Knowles+and+Philipp+Koehn%2E+2016%2E+Neural+interactive+translation+prediction%2E+In+Proceedings+of+the+Association+for+Machine+Translation+in+the+Americas%2C+pages+107%E2%80%93120%2E)
16. Philippe Laban, Andrew Hsi, John Canny, and Marti A. Hearst. 2020. The summary loop: Learning to write abstractive summaries without examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5135–5150, Online. Association for Computational Linguistic… [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Andrew+Hsi%2C+John+Canny%2C+and+Marti+A%2E+Hearst%2E+2020%2E+The+summary+loop%3A+Learning+to+write+abstractive+summaries+without+examples%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+Associat)
17. Haejun Lee, Drew A. Hudson, Kangwook Lee, and Christopher D. Manning. 2020. SLM: Learning a discourse language representation with sentence unshuffling. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1551–1562, Online. Association for Computa… [find](https://scholar.google.com/scholar?q=Haejun+Lee%2C+Drew+A%2E+Hudson%2C+Kangwook+Lee%2C+and+Christopher+D%2E+Manning%2E+2020%2E+SLM%3A+Learning+a+discourse+language+representation+with+sentence+unshuffling%2E+In+Proceedings+of+the+2020+Conference+on+Empiri)
18. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. [BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension](https://md.research.elamur.ai/free/en/papers/lewis-2020-bart-denoising-sequence-sequence-pre/). In Proceedings of the 58th Annual Meeting of the…
19. Chin-Yew Lin. 2004. [ROUGE: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
20. Stephen Mayhew, Gupta Nitish, and Dan Roth. 2020. Robust named entity recognition with truecasing pre-training. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8480–8487. [find](https://scholar.google.com/scholar?q=Stephen+Mayhew%2C+Gupta+Nitish%2C+and+Dan+Roth%2E+2020%2E+Robust+named+entity+recognition+with+truecasing+pre%2Dtraining%2E+Proceedings+of+the+AAAI+Conference+on+Artificial+Intelligence%2C+34%2805%29%3A8480%E2%80%938487%2E)
21. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797–1807, Brussels, Belgium. As… [find](https://scholar.google.com/scholar?q=Shashi+Narayan%2C+Shay+B%2E+Cohen%2C+and+Mirella+Lapata%2E+2018%2E+Don%E2%80%99t+give+me+the+details%2C+just+the+summary%21+topic%2Daware+convolutional+neural+networks+for+extreme+summarization%2E+In+Proceedings+of+the+2018+Co)
22. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS Autodiff Workshop. [find](https://scholar.google.com/scholar?q=Adam+Paszke%2C+Sam+Gross%2C+Soumith+Chintala%2C+Gregory+Chanan%2C+Edward+Yang%2C+Zachary+DeVito%2C+Zeming+Lin%2C+Alban+Desmaison%2C+Luca+Antiga%2C+and+Adam+Lerer%2E+2017%2E+Automatic+differentiation+in+PyTorch%2E+In+NIPS+Aut)
23. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. [Deep contextualized word representations](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
24. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan…
25. Martin F. Porter. 1997. An Algorithm for Suffix Stripping, page 313–316. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. [find](https://scholar.google.com/scholar?q=Martin+F%2E+Porter%2E+1997%2E+An+Algorithm+for+Suffix+Stripping%2C+page+313%E2%80%93316%2E+Morgan+Kaufmann+Publishers+Inc%2E%2C+San+Francisco%2C+CA%2C+USA%2E)
26. Kun Qian and Zhou Yu. 2019. Domain adaptive dialog generation via meta learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2639–2649, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kun+Qian+and+Zhou+Yu%2E+2019%2E+Domain+adaptive+dialog+generation+via+meta+learning%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+2639%E2%80%932649%2C+Florence%2C+)
27. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
28. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2E)
29. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Journal of Machine Learning Research, 21(140):1–67.
30. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Alexander+M%2E+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language)
31. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial winograd schema challenge at scale. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Keisuke+Sakaguchi%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+WinoGrande%3A+An+adversarial+winograd+schema+challenge+at+scale%2E+In+Proceedings+of+the+Thirty%2DFourth+AAAI+Conference+on+Artif)
32. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. [Automatically identifying words that can serve as labels for few-shot text classification](https://md.research.elamur.ai/free/en/papers/schick-2020-automatically-identifying-words-that-can/). In Proceedings of the 28th International Conference on Computational Linguistics, pages 5569–5578, Barcelona, Spain (Online). International Committee on C…
33. Timo Schick and Hinrich Schütze. 2020. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Sch%C3%BCtze%2E+2020%2E+Rare+words%3A+A+major+problem+for+contextualized+embeddings+and+how+to+fix+it+by+attentive+mimicking%2E+In+Proceedings+of+the+Thirty%2DFourth+AAAI+Conference+on+Artifi)
34. Timo Schick and Hinrich Schütze. 2021a. [Exploiting cloze questions for few shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, Kyiv, Ukraine (Online). International Committee on Compu…
35. Timo Schick and Hinrich Schütze. 2021b. [It’s not just size that matters: Small language models are also few-shot learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/). In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339–2352, Online. Assoc…
36. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 4596–4604. PMLR. [find](https://scholar.google.com/scholar?q=Noam+Shazeer+and+Mitchell+Stern%2E+2018%2E+Adafactor%3A+Adaptive+learning+rates+with+sublinear+memory+cost%2E+In+Proceedings+of+the+35th+International+Conference+on+Machine+Learning%2C+volume+80+of+Proceedings+)
37. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. 2016. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2818–2826. [find](https://scholar.google.com/scholar?q=C%2E+Szegedy%2C+V%2E+Vanhoucke%2C+S%2E+Ioffe%2C+J%2E+Shlens%2C+and+Z%2E+Wojna%2E+2016%2E+Rethinking+the+inception+architecture+for+computer+vision%2E+In+2016+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition+%28CVPR%29%2C)
38. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. [oLMpics-on what language model pre-training captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/). Transactions of the Association for Computational Linguistics, 8:743–758.
39. Joern Wuebker, Spence Green, John DeNero, Saša Hasan, and Minh-Thang Luong. 2016. Models and inference for prefix-constrained machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Berlin, Germany. Associa… [find](https://scholar.google.com/scholar?q=Joern+Wuebker%2C+Spence+Green%2C+John+DeNero%2C+Sa%C5%A1a+Hasan%2C+and+Minh%2DThang+Luong%2E+2016%2E+Models+and+inference+for+prefix%2Dconstrained+machine+translation%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Asso)
40. Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. 2020. [Unsupervised data augmentation for consistency training](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/). In Advances in Neural Information Processing Systems, volume 33, pages 6256–6268. Curran Associates, Inc.
41. Zhiquan Ye, Yuxia Geng, Jiaoyan Chen, Jingmin Chen, Xiaoxiao Xu, SuHang Zheng, Feng Wang, Jun Zhang, and Huajun Chen. 2020. Zero-shot text classification via reinforced self-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3014–3024, Online.… [find](https://scholar.google.com/scholar?q=Zhiquan+Ye%2C+Yuxia+Geng%2C+Jiaoyan+Chen%2C+Jingmin+Chen%2C+Xiaoxiao+Xu%2C+SuHang+Zheng%2C+Feng+Wang%2C+Jun+Zhang%2C+and+Huajun+Chen%2E+2020%2E+Zero%2Dshot+text+classification+via+reinforced+self%2Dtraining%2E+In+Proceedings+o)
42. Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. [PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization](https://md.research.elamur.ai/free/en/papers/zhang-2019-pegasus-pre-training-extracted-gap/). In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11328–11339, Virtual…
43. Rui Zhang and Joel Tetreault. 2019. This email could save your life: Introducing the task of email subject line generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 446–456, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rui+Zhang+and+Joel+Tetreault%2E+2019%2E+This+email+could+save+your+life%3A+Introducing+the+task+of+email+subject+line+generation%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computation)
44. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc.
45. Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao. 2019. [Does it make sense? And why? A pilot study for sense making and explanation](https://md.research.elamur.ai/free/en/papers/wang-2019-does-it-make-sense-why/). In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4020–4026, Florence, Italy. Association for Comput…
46. [Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)

*Updated 24 October 2026*