--- title: "Improving and Simplifying Pattern Exploiting Training — Tam 2021" description: "The authors present ADAPET, which drops the unlabeled data and instead extracts more… 0.5 points. 3 references, cited by 73 papers here." authors: ["Derek Tam", "Rakesh R. Menon", "Mohit Bansal", "Shashank Srivastava", "Colin Raffel"] year: 2021 venue: "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing" source: "https://aclanthology.org/2021.emnlp-main.407/" doi: "10.18653/v1/2021.emnlp-main.407" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/tam-2021-improving-simplifying-pattern-exploiting-training/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/tam-2021-improving-simplifying-pattern-exploiting-training/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Improving and Simplifying Pattern Exploiting Training # Improving and Simplifying Pattern Exploiting Training **Authors:** [Derek Tam](https://md.research.elamur.ai/free/en/authors/derek-tam/), Rakesh R. Menon, [Mohit Bansal](https://md.research.elamur.ai/free/en/authors/mohit-bansal/), [Shashank Srivastava](https://md.research.elamur.ai/free/en/authors/shashank-srivastava/) and [Colin Raffel](https://md.research.elamur.ai/free/en/authors/colin-raffel/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2021](https://md.research.elamur.ai/free/en/venues/emnlp/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2021.emnlp-main.407/) · [PDF](https://aclanthology.org/2021.emnlp-main.407.pdf) · [DOI 10.18653/v1/2021.emnlp-main.407](https://doi.org/10.18653/v1/2021.emnlp-main.407) · [OpenAlex](https://openalex.org/W3137573489) **Citations:** 111 citations (OpenAlex, 2026-10-14) · [73](https://md.research.elamur.ai/free/en/papers/tam-2021-improving-simplifying-pattern-exploiting-training/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Large language models such as GPT-3 can handle tasks from only a handful of labeled examples, but their size makes them hard to study or deploy. PET (Pattern Exploiting Training) closes much of this gap with a smaller ALBERT model by turning tasks into cloze questions. However, it also depends on thousands of task-specific unlabeled examples, which can be costly to collect for benchmarks like SuperGLUE. The authors present ADAPET, which drops the unlabeled data and instead extracts more training signal from the labeled set. It combines two changes to PET's objective: - **Decoupled label loss:** probabilities are normalized over the entire vocabulary, and a binary cross-entropy term promotes the correct label token while suppressing the incorrect ones. The same formulation covers single-token, WSC-style, and multi-token labels. - **Label-conditioned MLM:** the model reconstructs masked input tokens only when the input is paired with the correct label. With just 32 labeled examples per SuperGLUE task, ADAPET surpasses PET, iPET, and GPT-3 on the hidden test set. This shows that a denser objective can replace extra unlabeled data in few-shot learning. ## Key points - Setup: SuperGLUE with 32 labeled examples per task, using the same split, pre-trained ALBERT model, and hyperparameters as PET, but trained for 1k batches. Code is available at github.com/rrmenon10/ADAPET. - Test-set average: ADAPET reaches 76.0, compared with iPET 75.4, PET 74.0, and GPT-3 71.8. iPET and PET also use roughly 9K unlabeled examples per task. - Dev-set average: ADAPET reaches 77.3, about 0.5 points above iPET (76.8). It also beats the single-pattern baseline sPET (74.2) and GPT-3 (73.2). - Ablation: removing label conditioning sharply lowers CB from 91.1/88.1 to 75.0/62.8 (Acc./F1) and RTE from 75.1 to 64.3. - The decoupled loss replaces PET's hinge loss for multi-token labels (COPA, ReCoRD) by splitting the loss into per-token terms, which avoids normalization problems. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [University of North Carolina at Chapel Hill](https://md.research.elamur.ai/free/en/universities/university-of-north-carolina-at-chapel-hill/) — University, Chapel Hill, United States · 37 works · 14,332 citations - [Toyota Technological Institute at Chicago](https://md.research.elamur.ai/free/en/universities/toyota-technological-institute-at-chicago/) — University, Chicago, United States · 29 works · 10,273 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works ## Cite Cite as: Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava and Colin Raffel. 2021. Improving and Simplifying Pattern Exploiting Training. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{tam2021improving, title = {{Improving and Simplifying Pattern Exploiting Training}}, author = {Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava and Colin Raffel}, booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing}, year = {2021}, doi = {10.18653/v1/2021.emnlp-main.407}, url = {https://doi.org/10.18653/v1/2021.emnlp-main.407} } ``` - [← previous We Need to Consider Disagreement in Evaluation](https://md.research.elamur.ai/free/en/papers/basile-2021-we-need-consider-disagreement-evaluation/) - [next → GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning](https://md.research.elamur.ai/free/en/papers/chen-2021-geoqa-geometric-question-answering-benchmark/) - [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) ## References 3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. [Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri…](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/) 2. Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147– 2157, Online. Association for Computational L… [find](https://scholar.google.com/scholar?q=Jiaao+Chen%2C+Zichao+Yang%2C+and+Diyi+Yang%2E+2020%2E+Mix%2DText%3A+Linguistically%2Dinformed+interpolation+of+hidden+space+for+semi%2Dsupervised+text+classification%2E+In+Proceedings+of+the+58th+Annual+Meeting+of+the+) 3. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. [BoolQ: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… *Updated 24 October 2026*