--- title: "Generate, Annotate, and Learn: NLP with Synthetic Text (2022)" description: "The authors propose GAL (\"generate, annotate, and learn\"), a framework that uses language models as… 1 reference, cited by 31 papers here." authors: ["Xuanli He", "Islam Nassar", "Jamie Kiros", "Gholamreza Haffari", "Mohammad Norouzi"] year: 2022 venue: "Transactions of the Association for Computational Linguistics" source: "https://aclanthology.org/2022.tacl-1.48/" doi: "10.1162/tacl_a_00492" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/he-2022-generate-annotate-learn-nlp-synthetic/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/he-2022-generate-annotate-learn-nlp-synthetic/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) / Generate, Annotate, and Learn: NLP with Synthetic Text # Generate, Annotate, and Learn: NLP with Synthetic Text **Authors:** [Xuanli He](https://md.research.elamur.ai/free/en/authors/xuanli-he/), [Islam Nassar](https://md.research.elamur.ai/free/en/authors/islam-nassar/), [Jamie Kiros](https://md.research.elamur.ai/free/en/authors/jamie-kiros/), [Gholamreza Haffari](https://md.research.elamur.ai/free/en/authors/gholamreza-haffari/) and [Mohammad Norouzi](https://md.research.elamur.ai/free/en/authors/mohammad-norouzi/) · **Year:** 2022 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2022](https://md.research.elamur.ai/free/en/venues/tacl/2022/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2022.tacl-1.48/) · [PDF](https://aclanthology.org/2022.tacl-1.48.pdf) · [DOI 10.1162/tacl\_a\_00492](https://doi.org/10.1162/tacl_a_00492) · [OpenAlex](https://openalex.org/W4287120901) **Citations:** 38 citations (OpenAlex, 2026-10-14) · [31](https://md.research.elamur.ai/free/en/papers/he-2022-generate-annotate-learn-nlp-synthetic/cited-by/) in our library, of the 149,439 papers whose bibliographies we read ## Summary Many NLP tasks, especially GLUE tasks with sentence-pair inputs, lack in-domain unlabeled text, which limits self-training and knowledge distillation (KD). The authors propose GAL ("generate, annotate, and learn"), a framework that uses language models as a source of synthetic task-specific inputs. They fine-tune GPT-2 on a dataset's inputs with the labels removed, or prompt a large LM with a few examples. They then sample new text without any prompt. The strongest available classifier assigns soft pseudo labels to this text. For few-shot learning, the LM itself supplies hard labels. Finally, a new model is trained on the real labeled data plus the pseudo-labeled data, with no filtering step. The paper motivates the approach through empirical and vicinal risk minimization. It also argues, in theory and in experiments, that unconditional generation followed by teacher annotation beats class-conditional generation of labeled text. GAL sets a new state of the art for 6-layer transformer distillation on the GLUE leaderboard and also improves self-training and few-shot learning. This makes it a simple, widely applicable way to augment data. ## Key points - **Generation setup:** GPT-2 is fine-tuned without labels. Sentence pairs are joined with a [SEP] token, and generation starts from a [BOS] token using top-k sampling. Self-training runs use RoBERTa-base with 40× synthetic data. - **Gains over RoBERTa-base on GLUE dev sets (with GPT-2-large):** - SST-2: 94.8 → 95.3 - RTE: 78.8 → 81.4 - MRPC: 90.1 → 91.7 - CoLA: 63.6 → 65.1 Larger GPT-2 models mainly help on MRPC and CoLA. - **Soft vs. hard pseudo labels:** Soft labels outperform hard labels on all four tasks, for example 65.1 vs. 63.0 on CoLA. - **Class-conditional generation:** When class-conditional GPT-2 output is used with its predefined labels, results fall below the baseline (CoLA 58.4, RTE 74.4). Re-annotating the same text with GAL brings it back on par with unconditional generation. - **Few-shot learning:** On four 4-shot tasks, GAL gives an average gain of 1.3% and outperforms GPT-3-6B. - **Limitations:** Only sentence-level tasks were studied, and no data filtering was used. Document-level and structured prediction tasks (e.g., MT) are left for future work. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [Monash University](https://md.research.elamur.ai/free/en/universities/monash-university/) — University, Melbourne, Australia · 60 works · 54,866 citations - [Google (Canada)](https://md.research.elamur.ai/free/en/companies/google-canada/) — Company, Waterloo, Canada · 5 works · 6,841 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works ## Cite Cite as: Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari and Mohammad Norouzi. 2022. Generate, Annotate, and Learn: NLP with Synthetic Text. Transactions of the Association for Computational Linguistics. The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @article{he2022generate, title = {{Generate, Annotate, and Learn: NLP with Synthetic Text}}, author = {Xuanli He, Islam Nassar, Jamie Kiros, Gholamreza Haffari and Mohammad Norouzi}, journal = {Transactions of the Association for Computational Linguistics}, year = {2022}, doi = {10.1162/tacl\_a\_00492}, url = {https://doi.org/10.1162/tacl_a_00492} } ``` - [← previous POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://md.research.elamur.ai/free/en/papers/liu-2022-politics-pretraining-same-story-article/) - [next → Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric into a Document-Level Metric](https://md.research.elamur.ai/free/en/papers/vernikos-2022-embarrassingly-easy-document-level-mt/) - [↑ back to papers of 2022](https://md.research.elamur.ai/free/en/papers/2022/page/4/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. \#\# Conclusion Steven Abney. 2004. Understanding the Yarowsky algorithm. Computational Linguistics, 30(3):365–395. https://doi.org/10 .1162/0891201041850876 [find](https://scholar.google.com/scholar?q=%5C%23%5C%23+Conclusion+Steven+Abney%2E+2004%2E+Understanding+the+Yarowsky+algorithm%2E+Computational+Linguistics%2C+30%283%29%3A365%E2%80%93395%2E+https%3A%2F%2Fdoi%2Eorg%2F10+%2E1162%2F0891201041850876) *Updated 31 October 2026*