---
title: "Automatically Identifying Words That Can Serve as Labels for"
description: "The authors propose PETAL, which derives verbalizers automatically from a small labeled set. 24 references, cited by 72 papers here."
authors: ["Timo Schick", "Helmut Schmid", "Hinrich Schütze"]
year: 2020
venue: "Proceedings of the 28th International Conference on Computational Linguistics"
source: "https://aclanthology.org/2020.coling-main.488/"
doi: "10.18653/v1/2020.coling-main.488"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/schick-2020-automatically-identifying-words-that-can/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/schick-2020-automatically-identifying-words-that-can/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/) / Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

# Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification

**Authors:** [Timo Schick](https://md.research.elamur.ai/free/en/authors/timo-schick/), [Helmut Schmid](https://md.research.elamur.ai/free/en/authors/helmut-schmid/) and [Hinrich Schütze](https://md.research.elamur.ai/free/en/authors/hinrich-schutze/) · **Year:** 2020 · **Venue:** [Proceedings of the 28th International Conference on Computational Linguistics](https://md.research.elamur.ai/free/en/venues/coling/) ([COLING 2020](https://md.research.elamur.ai/free/en/venues/coling/2020/)) · **Publisher:** [International Committee on Computational Linguistics (ICCL)](https://md.research.elamur.ai/free/en/publishers/international-committee-on-computational-linguistics-iccl/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.coling-main.488/) · [PDF](https://aclanthology.org/2020.coling-main.488.pdf) · [DOI 10.18653/v1/2020.coling-main.488](https://doi.org/10.18653/v1/2020.coling-main.488) · [OpenAlex](https://openalex.org/W3096580779)

**Citations:** 161 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/schick-2020-automatically-identifying-words-that-can/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Pattern-Exploiting Training (PET) tackles few-shot text classification by rewriting each input as a cloze-style prompt for a pretrained masked language model. The label is then inferred from the word the model predicts at the mask. This depends on a verbalizer, a hand-picked word for each label, and choosing one requires both task expertise and intuition about which words the model understands well. The authors propose PETAL, which derives verbalizers automatically from a small labeled set. Searching over all word-to-label assignments directly is infeasible, so they split the task into one-vs-rest subproblems and reweight examples to offset the resulting class imbalance. Candidate tokens are scored with a likelihood-ratio loss rather than plain cross entropy, which would otherwise reward words that are simply probable everywhere. Candidates are prefiltered by frequency and fit to the pattern, and several words can jointly represent a label (multi-verbalizers). With only 50 training examples on Yelp, AG's News, Yahoo Questions and MNLI, PETAL outperforms supervised fine-tuning and the earlier automatic verbalizer search (AVS). It comes close to hand-crafted verbalizers, reducing the expertise needed to apply PET.

## Key points
- Average accuracy across the four tasks with |T| = 50 labeled examples:
  - supervised baseline: 56.3
  - PET + random verbalizers: 57.2
  - PET + AVS: 62.8
  - PETAL (joint): 65.9
  - PETAL (sep): 66.4
  - PET with manual verbalizers: 69.1
- The largest gain over AVS is on MNLI: 62.4 for PETAL (sep) versus 52.6 for AVS, close to the manual upper bound of 63.9. PETAL beats AVS on 3 of the 4 tasks; on AG's News the two are nearly tied.
- Candidate vocabulary is built in two steps. First, take the 10,000 most frequent real-word tokens in the task's unlabeled data. Then keep, for each label, the 1,000 tokens that best fit the positive examples. This deliberately favors frequent words, which language models understand better.
- The only new hyperparameter is the number of verbalizations per label (n_v). Main experiments use n_v = 10. Results stay stable for values from 1 to 100, and n_v = 3 gives the best overall score.
- The implementation is released in the PET repository (github.com/timoschick/pet). Experiments also use 10,000·k unlabeled examples per task, but these serve only PET's distillation step, not the verbalizer search.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Ludwig-Maximilians-Universität München](https://md.research.elamur.ai/free/en/universities/ludwig-maximilians-universitat-munchen/) — University, Munich, Germany · 44 works · 55,193 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Timo Schick, Helmut Schmid and Hinrich Schütze. 2020. Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification. Proceedings of the 28th International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{schick2020automatically,
  title = {{Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification}},
  author = {Timo Schick, Helmut Schmid and Hinrich Schütze},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.488},
  url = {https://doi.org/10.18653/v1/2020.coling-main.488}
}
```

- [← previous Politeness Transfer: A Tag and Generate Approach](https://md.research.elamur.ai/free/en/papers/madaan-2020-politeness-transfer-tag-generate-approach/)
- [next → Breaking Through the 80% Glass Ceiling: Raising the State of the Art in Word Sense Disambiguation by Incorporating Knowledge Graph Information](https://md.research.elamur.ai/free/en/papers/bevilacqua-2020-breaking-through-80-glass-ceiling/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/3/)

## References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections. In Proceedings of the Fifth ACM Conference on Digital Libraries, DL ’00, page 85–94, New York, NY, USA. Association for Computing Machinery. [find](https://scholar.google.com/scholar?q=Eugene+Agichtein+and+Luis+Gravano%2E+2000%2E+Snowball%3A+Extracting+relations+from+large+plain%2Dtext+collections%2E+In+Proceedings+of+the+Fifth+ACM+Conference+on+Digital+Libraries%2C+DL+%E2%80%9900%2C+page+85%E2%80%9394%2C+New+York)
2. David S. Batista, Bruno Martins, and Mário J. Silva. 2015. Semi-supervised bootstrapping of relationship extractors with distributional semantics. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 499–504, Lisbon, Portugal, September. Association for C… [find](https://scholar.google.com/scholar?q=David+S%2E+Batista%2C+Bruno+Martins%2C+and+Ma%CC%81rio+J%2E+Silva%2E+2015%2E+Semi%2Dsupervised+bootstrapping+of+relationship+extractors+with+distributional+semantics%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+M)
3. Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert. 2020. Inducing relational knowledge from BERT. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Zied+Bouraoui%2C+Jose+Camacho%2DCollados%2C+and+Steven+Schockaert%2E+2020%2E+Inducing+relational+knowledge+from+BERT%2E+In+Proceedings+of+the+Thirty%2DFourth+AAAI+Conference+on+Artificial+Intelligence%2E)
4. Sergey Brin. 1999. Extracting patterns and relations from the world wide web. In Paolo Atzeni, Alberto Mendelzon, and Giansalvatore Mecca, editors, The World Wide Web and Databases, pages 172–183, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Sergey+Brin%2E+1999%2E+Extracting+patterns+and+relations+from+the+world+wide+web%2E+In+Paolo+Atzeni%2C+Alberto+Mendelzon%2C+and+Giansalvatore+Mecca%2C+editors%2C+The+World+Wide+Web+and+Databases%2C+pages+172%E2%80%93183%2C+Ber)
5. [Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/)
6. Joe Davison, Joshua Feldman, and Alexander Rush. 2019. [Commonsense knowledge mining from pretrained models](https://md.research.elamur.ai/free/en/papers/davison-2019-commonsense-knowledge-mining-pretrained-models/). In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1173–1…
7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
8. Allyson Ettinger. 2020. [What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models](https://md.research.elamur.ai/free/en/papers/ettinger-2020-what-bert-is-not-lessons/). Transactions of the Association for Computational Linguistics, 8:34–48, Jan.
9. Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2019. [How can we know what language models know?](https://md.research.elamur.ai/free/en/papers/jiang-2020-how-can-we-know-what/) Computing Research Repository, arXiv:1911.12543.
10. Yoonkyung Lee, Yi Lin, and Grace Wahba. 2001. Multicategory support vector machines. Technical report, Department of Statistics, University of Madison, Wisconsin. [find](https://scholar.google.com/scholar?q=Yoonkyung+Lee%2C+Yi+Lin%2C+and+Grace+Wahba%2E+2001%2E+Multicategory+support+vector+machines%2E+Technical+report%2C+Department+of+Statistics%2C+University+of+Madison%2C+Wisconsin%2E)
11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository, arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+C)
12. Juri Opitz. 2019. Argumentative relation classification as plausibility ranking. In Preliminary proceedings of the 15th Conference on Natural Language Processing (KONVENS 2019): Long Papers, pages 193–202, Erlangen, Germany. German Society for Computational Linguistics & Language Technology. [find](https://scholar.google.com/scholar?q=Juri+Opitz%2E+2019%2E+Argumentative+relation+classification+as+plausibility+ranking%2E+In+Preliminary+proceedings+of+the+15th+Conference+on+Natural+Language+Processing+%28KONVENS+2019%29%3A+Long+Papers%2C+pages+193)
13. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. [Language models as knowledge bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La…
14. Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. Computing Research Repository, arXiv:1912.10165. — [arXiv:1912.10165](https://arxiv.org/abs/1912.10165) [find](https://scholar.google.com/scholar?q=Raul+Puri+and+Bryan+Catanzaro%2E+2019%2E+Zero%2Dshot+text+classification+with+generative+language+models%2E+Computing+Research+Repository%2C+arXiv%3A1912%2E10165%2E+%E2%80%94+%5BarXiv%3A1912%2E10165%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1912%2E101)
15. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
16. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeff+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2E)
17. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). Computing Research Repository, arXiv:1910.10683.
18. Timo Schick and Hinrich Schütze. 2020a. [Exploiting cloze questions for few shot text classification and natural language inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). Computing Research Repository, arXiv:2001.07676.
19. Timo Schick and Hinrich Schütze. 2020b. Rare words: A major problem for contextualized embeddings and how to fix it by attentive mimicking. In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence. [find](https://scholar.google.com/scholar?q=Timo+Schick+and+Hinrich+Schu%CC%88tze%2E+2020b%2E+Rare+words%3A+A+major+problem+for+contextualized+embeddings+and+how+to+fix+it+by+attentive+mimicking%2E+In+Proceedings+of+the+Thirty%2DFourth+AAAI+Conference+on+Arti)
20. Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Unsupervised commonsense question answering with self-talk. Computing Research Repository, arXiv:2004.05483. [find](https://scholar.google.com/scholar?q=Vered+Shwartz%2C+Peter+West%2C+Ronan+Le+Bras%2C+Chandra+Bhagavatula%2C+and+Yejin+Choi%2E+2020%2E+Unsupervised+commonsense+question+answering+with+self%2Dtalk%2E+Computing+Research+Repository%2C+arXiv%3A2004%2E05483%2E)
21. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. [oLMpics – on what language model pre-training captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/). Computing Research Repository, arXiv:1912.13283.
22. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)
23. Hengshuai Yao, Dong-lai Zhu, Bei Jiang, and Peng Yu. 2020. Negative log likelihood ratio loss for deep neural network classification. In Kohei Arai, Rahul Bhatia, and Supriya Kapoor, editors, Proceedings of the Future Technologies Conference (FTC) 2019, pages 276–282, Cham. Springer International Pu… [find](https://scholar.google.com/scholar?q=Hengshuai+Yao%2C+Dong%2Dlai+Zhu%2C+Bei+Jiang%2C+and+Peng+Yu%2E+2020%2E+Negative+log+likelihood+ratio+loss+for+deep+neural+network+classification%2E+In+Kohei+Arai%2C+Rahul+Bhatia%2C+and+Supriya+Kapoor%2C+editors%2C+Proceedi)
24. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 649–657. Curran Associates, Inc.

*Updated 24 October 2026*