---
title: "Self-training Improves Pre-training for Natural Language"
description: "It builds a query embedding for each task from the labeled training set and uses it to… 73.5 Pearson. 57 references, cited by 63 papers here."
authors: ["Jingfei Du", "Edouard Grave", "Beliz Gunel", "Vishrav Chaudhary", "Onur Celebi", "Michael Auli", "Veselin Stoyanov", "Alexis Conneau"]
year: 2021
venue: "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/2021.naacl-main.426/"
doi: "10.18653/v1/2021.naacl-main.426"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/du-2021-self-training-improves-pre-training/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/du-2021-self-training-improves-pre-training/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/) / Self-training Improves Pre-training for Natural Language Understanding

# Self-training Improves Pre-training for Natural Language Understanding

**Authors:** [Jingfei Du](https://md.research.elamur.ai/free/en/authors/jingfei-du/), [Edouard Grave](https://md.research.elamur.ai/free/en/authors/edouard-grave/), [Beliz Gunel](https://md.research.elamur.ai/free/en/authors/beliz-gunel/), [Vishrav Chaudhary](https://md.research.elamur.ai/free/en/authors/vishrav-chaudhary/), [Onur Celebi](https://md.research.elamur.ai/free/en/authors/onur-celebi/), [Michael Auli](https://md.research.elamur.ai/free/en/authors/michael-auli/), [Veselin Stoyanov](https://md.research.elamur.ai/free/en/authors/veselin-stoyanov/) and [Alexis Conneau](https://md.research.elamur.ai/free/en/authors/alexis-conneau/) · **Year:** 2021 · **Venue:** [Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/naacl/) ([NAACL 2021](https://md.research.elamur.ai/free/en/venues/naacl/2021/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2021.naacl-main.426/) · [PDF](https://aclanthology.org/2021.naacl-main.426.pdf) · [DOI 10.18653/v1/2021.naacl-main.426](https://doi.org/10.18653/v1/2021.naacl-main.426) · [OpenAlex](https://openalex.org/W3168921237)

**Citations:** 106 citations (OpenAlex, 2026-10-14) · [63](https://md.research.elamur.ai/free/en/papers/du-2021-self-training-improves-pre-training/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary

Self-training uses a teacher model to label unannotated examples for a student model. It usually assumes that unlabeled text from the target task's domain is already available, and that assumption breaks down for many low-resource tasks. The authors ask whether self-training still helps on top of strong language-model pre-training, and how to obtain suitable unlabeled data in the first place. Their answer is SentAugment. It builds a query embedding for each task from the labeled training set and uses it to retrieve similar sentences from a CommonCrawl-derived bank of billions of web sentences. Retrieval relies on a paraphrastic sentence encoder (SASE) trained with a triplet loss.

A fine-tuned RoBERTa-Large teacher then labels the retrieved sentences. Only the most confident examples per class are kept, with the original label ratio preserved. A new student is trained on these soft labels using KL-divergence. Self-training gives consistent gains over RoBERTa-Large, and the method also benefits knowledge distillation and few-shot learning. This suggests that pre-training and self-training are complementary, and that open web data can stand in for in-domain corpora.

## Key points

- Self-training with SentAugment lifts RoBERTa-Large by 1.2% accuracy on average across six benchmarks, with gains of up to 2.6% on individual text classification tasks:
  - SST-2 and SST-5
  - CR
  - IMP (hate speech)
  - TREC
  - CoNLL NER
- In few-shot settings, accuracy rises by 3.5% on average. For knowledge distillation into a much smaller student, the distilled model improves by 2.9%.
- Three query types are compared: all-average, label-average and per-sentence. Scoring 1B sentences with label-average queries takes about one minute.
- Growing the sentence bank from 50M to 1B sentences brings clear gains, but these level off between 1B and 5B.
- SASE (256-dim) beats earlier encoders such as Sentence-BERT, uSIF and BGT:
  - on STS 2012–2016, it averages 73.5 Pearson r×100, a gain of 0.2;
  - on STS-B, it scores 80.8, a gain of 0.9.
- The code, models and nearest-neighbor indexes are released at github.com/facebookresearch/SentAugment.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,343 works

## Cite

Cite as: Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov and Alexis Conneau. 2021. Self-training Improves Pre-training for Natural Language Understanding. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{du2021self,
  title = {{Self-training Improves Pre-training for Natural Language Understanding}},
  author = {Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov and Alexis Conneau},
  booktitle = {Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-main.426},
  url = {https://doi.org/10.18653/v1/2021.naacl-main.426}
}
```

- [← previous Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain](https://md.research.elamur.ai/free/en/papers/freitag-2021-results-wmt21-metrics-shared-task/)
- [next → Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing](https://md.research.elamur.ai/free/en/papers/nguyen-2021-trankit-light-weight-transformer-based/)
- [↑ back to papers of 2021](https://md.research.elamur.ai/free/en/papers/2021/page/3/)

## References

57 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015. [Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability](https://md.research.elamur.ai/free/en/papers/agirre-2015-semeval-2015-task-2-semantic/). In SemEval@ NAACL-HLT, pages 2…
2. Eneko Agirre, Carmen Baneab, Claire Cardiec, Daniel Cerd, Mona Diabe, Aitor Gonzalez-Agirre, Weiwei Guof, Rada Mihalceab, German Rigaua, and Janyce Wiebeg. 2014. [Semeval-2014 task 10: Multilingual semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2014-semeval-2014-task-10-multilingual/).
3. Eneko Agirre, Carmen Baneab, Daniel Cerd, Mona Diabe, Aitor Gonzalez-Agirre, Rada Mihalceab, German Rigaua, Janyce Wiebef, and Basque Country Donostia. 2016. [Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation](https://md.research.elamur.ai/free/en/papers/agirre-2016-semeval-2016-task-1-semantic/). Proceedings of SemEval, pages 497–511.
4. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-agirre, and Weiwei Guo. 2013. [sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity](https://md.research.elamur.ai/free/en/papers/agirre-2013-sem-2013-shared-task-semantic/). In In *SEM 2013: The Second Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics.
5. Eneko Agirre, Mona Diab, Daniel Cer, and Aitor Gonzalez-Agirre. 2012. [Semeval-2012 task 6: A pilot on semantic textual similarity](https://md.research.elamur.ai/free/en/papers/agirre-2012-semeval-2012-task-6-pilot/). In Proceedings of Semeval-2012, pages 385–393.
6. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In 5th International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Sanjeev+Arora%2C+Yingyu+Liang%2C+and+Tengyu+Ma%2E+2017%2E+A+simple+but+tough%2Dto%2Dbeat+baseline+for+sentence+embeddings%2E+In+5th+International+Conference+on+Learning+Representations%2E)
7. Mikel Artetxe and Holger Schwenk. 2019. [Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/). Transactions of the Association for Computational Linguistics, 7:597–610.
8. Ondrej Bojar and Ales Tamchyna. 2011. Improving translation model by monolingual data. In Workshop on Statistical Machine Translation (WMT). [find](https://scholar.google.com/scholar?q=Ondrej+Bojar+and+Ales+Tamchyna%2E+2011%2E+Improving+translation+model+by+monolingual+data%2E+In+Workshop+on+Statistical+Machine+Translation+%28WMT%29%2E)
9. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. [Language models are few-shot learners](https://md.research.elamur.ai/free/en/papers/brown-2020-language-models-few-shot-learners/). arXiv preprint arXiv:2005.14165.
10. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. [Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/). arXiv preprint arXiv:1708.00055.
11. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. [Universal sentence encoder for english](https://md.research.elamur.ai/free/en/papers/cer-2018-universal-sentence-encoder-english/).
12. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169–174. [find](https://scholar.google.com/scholar?q=In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%3A+System+Demonstrations%2C+pages+169%E2%80%93174%2E)
13. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. [Supervised learning of universal sentence representations from natural language inference data](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/). In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,…
14. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. NeurIPS. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+NeurIPS%2E)
15. Mathias Johan Philip Creutz et al. 2018. Open subtitles paraphrase corpus for six languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Mathias+Johan+Philip+Creutz+et+al%2E+2018%2E+Open+subtitles+paraphrase+corpus+for+six+languages%2E+In+Proceedings+of+the+Eleventh+International+Conference+on+Language+Resources+and+Evaluation+%28LREC+2018%29%2E+E)
16. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3079%E2%80%933087%2E)
17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). NAACL.
18. William B Dolan and Chris Brockett. 2005. [Automatically constructing a corpus of sentential paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the International Workshop on Paraphrasing.
19. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Sergey+Edunov%2C+Myle+Ott%2C+Michael+Auli%2C+and+David+Grangier%2E+2018%2E+Understanding+back%2Dtranslation+at+scale%2E+In+Proc%2E+of+EMNLP%2E)
20. Kawin Ethayarajh. 2018a. Unsupervised random walk sentence embeddings: A strong but simple baseline. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2018a%2E+Unsupervised+random+walk+sentence+embeddings%3A+A+strong+but+simple+baseline%2E)
21. Kawin Ethayarajh. 2018b. Unsupervised random walk sentence embeddings: A strong but simple baseline. In Proceedings of The Third Workshop on Representation Learning for NLP. [find](https://scholar.google.com/scholar?q=Kawin+Ethayarajh%2E+2018b%2E+Unsupervised+random+walk+sentence+embeddings%3A+A+strong+but+simple+baseline%2E+In+Proceedings+of+The+Third+Workshop+on+Representation+Learning+for+NLP%2E)
22. Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato. 2019. Revisiting self-training for neural sequence generation. arXiv preprint arXiv:1909.13788. [find](https://scholar.google.com/scholar?q=Junxian+He%2C+Jiatao+Gu%2C+Jiajun+Shen%2C+and+Marc%E2%80%99Aurelio+Ranzato%2E+2019%2E+Revisiting+self%2Dtraining+for+neural+sequence+generation%2E+arXiv+preprint+arXiv%3A1909%2E13788%2E)
23. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. [Distilling the knowledge in a neural network](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/). arXiv preprint arXiv:1503.02531.
24. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 328–339. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+Computational+Linguistics+%28Volum)
25. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of SIGKDD, pages 168–177. [find](https://scholar.google.com/scholar?q=Minqing+Hu+and+Bing+Liu%2E+2004%2E+Mining+and+summarizing+customer+reviews%2E+In+Proceedings+of+SIGKDD%2C+pages+168%E2%80%93177%2E)
26. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. [Billion-scale similarity search with gpus](https://md.research.elamur.ai/free/en/papers/johnson-2017-billion-scale-similarity-search-gpus/). IEEE Transactions on Big Data.
27. Jacob Kahn, Ann Lee, and Awni Hannun. 2020. Self-training for end-to-end speech recognition. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7084–7088. IEEE. [find](https://scholar.google.com/scholar?q=Jacob+Kahn%2C+Ann+Lee%2C+and+Awni+Hannun%2E+2020%2E+Self%2Dtraining+for+end%2Dto%2Dend+speech+recognition%2E+In+ICASSP+2020%2D2020+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%28ICASSP%29%2C+page)
28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. [Exploring the limits of transfer learning with a unified text-to-text transformer](https://md.research.elamur.ai/free/en/papers/raffel-2019-exploring-limits-transfer-learning-unified/). arXiv preprint arXiv:1910.10683.
29. Nils Reimers and Iryna Gurevych. 2019. [Sentence-bert: Sentence embeddings using siamese bert-networks](https://md.research.elamur.ai/free/en/papers/reimers-2019-sentence-bert-sentence-embeddings-siamese/). arXiv preprint arXiv:1908.10084.
30. Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Russ+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+pages+)
31. Erik F Sang and Fien De Meulder. 2003. [Introduction to the conll-2003 shared task: Language-independent named entity recognition](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/). arXiv preprint cs/0306050.
32. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT Summit. [find](https://scholar.google.com/scholar?q=Philipp+Koehn%2E+2005%2E+Europarl%3A+A+parallel+corpus+for+statistical+machine+translation%2E+In+MT+Summit%2E)
33. H Scudder. 1965. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371. [find](https://scholar.google.com/scholar?q=H+Scudder%2E+1965%2E+Probability+of+error+of+some+adaptive+pattern%2Drecognition+machines%2E+IEEE+Transactions+on+Information+Theory%2C+11%283%29%3A363%E2%80%93371%2E)
34. Pierre Lison, Jörg Tiedemann, Milen Kouylekov, et al. 2019. Open subtitles 2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In LREC 2018, Eleventh International Conference on Language Resources and Evaluation. European Language Resources Association (ELRA). [find](https://scholar.google.com/scholar?q=Pierre+Lison%2C+Jo%CC%88rg+Tiedemann%2C+Milen+Kouylekov%2C+et+al%2E+2019%2E+Open+subtitles+2018%3A+Statistical+rescoring+of+sentence+alignments+in+large%2C+noisy+parallel+corpora%2E+In+LREC+2018%2C+Eleventh+International+Co)
35. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
36. David McClosky, Eugene Charniak, and Mark Johnson. 2006. [Effective self-training for parsing](https://md.research.elamur.ai/free/en/papers/mcclosky-2006-effective-self-training-parsing/). In Proc. of NAACL.
37. Tomáš Mikolov, Édouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. [Advances in pre-training distributed word representations](https://md.research.elamur.ai/free/en/papers/mikolov-2017-advances-pre-training-distributed-word/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
38. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. [Distributed representations of words and phrases and their compositionality](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/). In Advances in Neural Information Processing Systems.
39. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of NAACL-HLT 2019: Demonstrations.
40. Daniel S Park, Yu Zhang, Ye Jia, Wei Han, Chung-Cheng Chiu, Bo Li, Yonghui Wu, and Quoc V Le. 2020. Improved noisy student training for automatic speech recognition. arXiv preprint arXiv:2005.09629. — [arXiv:2005.09629](https://arxiv.org/abs/2005.09629) [find](https://scholar.google.com/scholar?q=Daniel+S+Park%2C+Yu+Zhang%2C+Ye+Jia%2C+Wei+Han%2C+Chung%2DCheng+Chiu%2C+Bo+Li%2C+Yonghui+Wu%2C+and+Quoc+V+Le%2E+2020%2E+Improved+noisy+student+training+for+automatic+speech+recognition%2E+arXiv+preprint+arXiv%3A2005%2E09629%2E+%E2%80%94)
41. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E+arXiv%2E)
42. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709. [find](https://scholar.google.com/scholar?q=Rico+Sennrich%2C+Barry+Haddow%2C+and+Alexandra+Birch%2E+2015%2E+Improving+neural+machine+translation+models+with+monolingual+data%2E+arXiv+preprint+arXiv%3A1511%2E06709%2E)
43. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631–1642. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
44. Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. 2018. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv:1804.00079. — [arXiv:1804.00079](https://arxiv.org/abs/1804.00079) [find](https://scholar.google.com/scholar?q=Sandeep+Subramanian%2C+Adam+Trischler%2C+Yoshua+Bengio%2C+and+Christopher+J+Pal%2E+2018%2E+Learning+general+purpose+distributed+sentence+representations+via+large+scale+multi%2Dtask+learning%2E+arXiv+preprint+arXiv)
45. Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Edouard Grave, Tatiana Likhomanenko, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2019. End-to-end asr: from supervised to semi-supervised learning with modern architectures. arXiv preprint arXiv:1911.08460. [find](https://scholar.google.com/scholar?q=Gabriel+Synnaeve%2C+Qiantong+Xu%2C+Jacob+Kahn%2C+Edouard+Grave%2C+Tatiana+Likhomanenko%2C+Vineel+Pratap%2C+Anuroop+Sriram%2C+Vitaliy+Liptchinsky%2C+and+Ronan+Collobert%2E+2019%2E+End%2Dto%2Dend+asr%3A+from+supervised+to+semi%2Ds)
46. Ellen M Voorhees and Dawn M Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207. ACM. [find](https://scholar.google.com/scholar?q=Ellen+M+Voorhees+and+Dawn+M+Tice%2E+2000%2E+Building+a+question+answering+test+collection%2E+In+Proceedings+of+the+23rd+annual+international+ACM+SIGIR+conference+on+Research+and+development+in+information+r)
47. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2019. [Ccnet: Extracting high quality monolingual datasets from web crawl data](https://md.research.elamur.ai/free/en/papers/wenzek-2020-ccnet-extracting-high-quality-monolingual/). arXiv preprint arXiv:1911.00359.
48. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. [Towards universal paraphrastic sentence embeddings](https://md.research.elamur.ai/free/en/papers/wieting-2015-towards-universal-paraphrastic-sentence-embeddings/). ICLR.
49. John Wieting and Kevin Gimpel. 2017. [Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations](https://md.research.elamur.ai/free/en/papers/wieting-2018-paranmt-50m-pushing-limits-paraphrastic/). arXiv preprint arXiv:1711.05732.
50. John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019a. Simple and effective paraphrastic similarity from parallel translations. arXiv preprint arXiv:1909.13872. [find](https://scholar.google.com/scholar?q=John+Wieting%2C+Kevin+Gimpel%2C+Graham+Neubig%2C+and+Taylor+Berg%2DKirkpatrick%2E+2019a%2E+Simple+and+effective+paraphrastic+similarity+from+parallel+translations%2E+arXiv+preprint+arXiv%3A1909%2E13872%2E)
51. John Wieting, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019b. A bilingual generative transformer for semantic sentence embedding. arXiv preprint arXiv:1911.03895. [find](https://scholar.google.com/scholar?q=John+Wieting%2C+Graham+Neubig%2C+and+Taylor+Berg%2DKirkpatrick%2E+2019b%2E+A+bilingual+generative+transformer+for+semantic+sentence+embedding%2E+arXiv+preprint+arXiv%3A1911%2E03895%2E)
52. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of the 2nd Workshop on Evaluating Vector-Space Representations for NLP. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+2017%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+Proceedings+of+the+2nd+Workshop+on+Evaluating+Vector%2DSpace+Rep)
53. Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2019. [Unsupervised data augmentation for consistency training](https://md.research.elamur.ai/free/en/papers/xie-2019-unsupervised-data-augmentation-consistency-training/). arXiv preprint arXiv:1904.12848.
54. Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. 2020. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10687–10698. [find](https://scholar.google.com/scholar?q=Qizhe+Xie%2C+Minh%2DThang+Luong%2C+Eduard+Hovy%2C+and+Quoc+V+Le%2E+2020%2E+Self%2Dtraining+with+noisy+student+improves+imagenet+classification%2E+In+Proceedings+of+the+IEEE%2FCVF+Conference+on+Computer+Vision+and+Patte)
55. I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546. — [arXiv:1905.00546](https://arxiv.org/abs/1905.00546) [find](https://scholar.google.com/scholar?q=I+Zeki+Yalniz%2C+Herve%CC%81+Je%CC%81gou%2C+Kan+Chen%2C+Manohar+Paluri%2C+and+Dhruv+Mahajan%2E+2019%2E+Billion%2Dscale+semi%2Dsupervised+learning+for+image+classification%2E+arXiv+preprint+arXiv%3A1905%2E00546%2E+%E2%80%94+%5BarXiv%3A1905%2E00546%5D%28)
56. David Yarowsky. 1995. [Unsupervised word sense disambiguation rivaling supervised methods](https://md.research.elamur.ai/free/en/papers/yarowsky-1995-unsupervised-word-sense-disambiguation-rivaling/). In 33rd annual meeting of the association for computational linguistics, pages 189–196.
57. Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D Cubuk, and Quoc V Le. 2020. Rethinking pre-training and self-training. arXiv preprint arXiv:2006.06882. — [arXiv:2006.06882](https://arxiv.org/abs/2006.06882) [find](https://scholar.google.com/scholar?q=Barret+Zoph%2C+Golnaz+Ghiasi%2C+Tsung%2DYi+Lin%2C+Yin+Cui%2C+Hanxiao+Liu%2C+Ekin+D+Cubuk%2C+and+Quoc+V+Le%2E+2020%2E+Rethinking+pre%2Dtraining+and+self%2Dtraining%2E+arXiv+preprint+arXiv%3A2006%2E06882%2E+%E2%80%94+%5BarXiv%3A2006%2E06882%5D%28http)

*Updated 24 October 2026*