--- title: "SSMBA — paper by Ng et al. (2020) — summary and ideas" description: "The authors propose SSMBA, a task-agnostic augmentation scheme based on denoising autoencoders. 3 references, cited by 60 papers here." authors: ["Nathan Ng", "Kyunghyun Cho", "Marzyeh Ghassemi"] year: 2020 venue: "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)" source: "https://aclanthology.org/2020.emnlp-main.97/" doi: "10.18653/v1/2020.emnlp-main.97" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/ng-2020-ssmba-self-supervised-manifold-based/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ng-2020-ssmba-self-supervised-manifold-based/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness # SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness **Authors:** [Nathan Ng](https://md.research.elamur.ai/free/en/authors/nathan-ng/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) and Marzyeh Ghassemi · **Year:** 2020 · **Venue:** [Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2020](https://md.research.elamur.ai/free/en/venues/emnlp/2020/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.emnlp-main.97/) · [PDF](https://aclanthology.org/2020.emnlp-main.97.pdf) · [DOI 10.18653/v1/2020.emnlp-main.97](https://doi.org/10.18653/v1/2020.emnlp-main.97) · [OpenAlex](https://openalex.org/W3089233166) **Citations:** 95 citations (OpenAlex, 2026-10-14) · [60](https://md.research.elamur.ai/free/en/papers/ng-2020-ssmba-self-supervised-manifold-based/cited-by/) in our library, of the 141,840 papers whose bibliographies we read ## Summary Classifiers and sequence models trained on one domain often perform poorly on out-of-domain (OOD) inputs. For images, simple transformations like rotation keep new examples realistic, but no comparable meaning-preserving operations exist for text. The authors propose SSMBA, a task-agnostic augmentation scheme based on denoising autoencoders. It works in two steps: 1. It corrupts a training input so that it leaves the data manifold, using the MLM masking procedure. 2. It restores the input with a pre-trained masked language model, which pulls it back onto the manifold. The new examples get their labels in one of two ways: they inherit the label of the source example, or a teacher model fit to the unaugmented training set assigns them. The method needs no task knowledge and no dataset-specific fine-tuning. It was tested on sentiment analysis, NLI and machine translation, covering 9 datasets and 4 model types. SSMBA beat both unaugmented baselines and competing augmentation methods, in-domain and OOD. This makes it a cheap, general tool for improving robustness, including in low-resource settings. ## Key points - **Headline gains:** +0.8% OOD accuracy on Amazon reviews, +1.8% OOD accuracy on MNLI, and +1.4 BLEU in-domain on IWSLT14 German-English. - **Benchmarks:** - Sentiment: AR-Full, AR-Clothing, Yelp, and Movies (SST2 paired with IMDb). - NLI: MNLI and ANLI (R1–R3). - Machine translation: IWSLT14 de→en, OPUS (train on medical, test on Koran, Ubuntu, subtitles and legal text), and de→rm (Allegra corpus plus Swiss press releases, tested on Convivenza blogs). - **Ablations on reconstruction and sampling:** - The choice of reconstruction model matters little: DistilRoBERTa (82M), RoBERTa BASE (125M) and RoBERTa LARGE (355M) give similar improvements. - Unrestricted sampling from the MLM outperforms top-k sampling. - **Ablations on corruption and data size:** - For sentiment, OOD accuracy is best at about 45% corrupted tokens, and augmented models still beat the baseline even at 95%. The NLI task, by contrast, is hurt by heavy corruption. - Generating more augmented sentences per example raises gains and lowers variance. - Benefits persist with fewer than 100 training examples. - **Code:** available at github.com/nng555/ssmba. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions - [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 136 works · 86,390 citations - [University of Toronto](https://md.research.elamur.ai/free/en/universities/university-of-toronto/) — University, Toronto, Canada · 47 works · 50,254 citations - [Vector Institute](https://md.research.elamur.ai/free/en/labs/vector-institute/) — Research institute, Toronto, Canada · 19 works · 4,281 citations - [Courant Institute of Mathematical Sciences](https://md.research.elamur.ai/free/en/universities/courant-institute-of-mathematical-sciences/) — University, New York, United States · 7 works · 8,082 citations 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works ## Cite Cite as: Nathan Ng, Kyunghyun Cho and Marzyeh Ghassemi. 2020. SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{ng2020ssmba, title = {{SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness}}, author = {Nathan Ng, Kyunghyun Cho and Marzyeh Ghassemi}, booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, year = {2020}, doi = {10.18653/v1/2020.emnlp-main.97}, url = {https://doi.org/10.18653/v1/2020.emnlp-main.97} } ``` - [← previous Reducing Quantity Hallucinations in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/zhao-2020-reducing-quantity-hallucinations-abstractive-summarization/) - [next → Fill in the BLANC: Human-free quality estimation of document summaries](https://md.research.elamur.ai/free/en/papers/vasilyev-2020-fill-blanc-human-free-quality/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) ## References 3 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. c l Train Valid Test † AR-Clothing Men Women Baby Shoes Jianmo Ni 2019 Jianmo Ni 2019 Jianmo Ni 2019 Jianmo Ni 2019 5 5 5 5 31 40 29 41 25k 25k † 25k † 25k † 2k 2k 2k 2k 2k 2k 2k 2k AR-Full Books Clothing, Shoes & Jewelry Home and Kitchen Kindle Store Movies & TV Pet Supplies Sports & Outdoors Elect… [find](https://scholar.google.com/scholar?q=c+l+Train+Valid+Test+%E2%80%A0+AR%2DClothing+Men+Women+Baby+Shoes+Jianmo+Ni+2019+Jianmo+Ni+2019+Jianmo+Ni+2019+Jianmo+Ni+2019+5+5+5+5+31+40+29+41+25k+25k+%E2%80%A0+25k+%E2%80%A0+25k+%E2%80%A0+2k+2k+2k+2k+2k+2k+2k+2k+AR%2DFull+Books+Clot) 2. Nie et al. 2019 Nie et al. 2019 Nie et al. 2019 3 3 3 92 90 82 17k 46k 100k 1k 1k 1k 1k 1k 1k IWSLT IWSLT Cettolo et al. 2014 - 160k 7k 7k OPUS Medical IT Koran Law Subtitles Tiedemann 2012 Tiedemann 2012 Tiedemann 2012 Tiedemann 2012 Tiedemann 2012 -- -- - 13 14 23 31 10 1.1m -- -- 2k 2k 2k 2k 2k 2… [find](https://scholar.google.com/scholar?q=Nie+et+al%2E+2019+Nie+et+al%2E+2019+Nie+et+al%2E+2019+3+3+3+92+90+82+17k+46k+100k+1k+1k+1k+1k+1k+1k+IWSLT+IWSLT+Cettolo+et+al%2E+2014+%2D+160k+7k+7k+OPUS+Medical+IT+Koran+Law+Subtitles+Tiedemann+2012+Tiedemann+) 3. AR-Full * * RNN CNN 50% 40% Unrestricted Sampling Unrestricted Sampling Preserve Label Soft Label 5 5 * * SST2 IMDb SST2 IMDb RNN CNN RNN RNN CNN CNN 60% 40% 10% 20% 60% 30% Unrestricted Sampling Unrestricted Sampling Unrestricted Sampling Unrestricted Sampling Unrestricted Sampling Unrestricted Sam… [find](https://scholar.google.com/scholar?q=AR%2DFull+%2A+%2A+RNN+CNN+50%25+40%25+Unrestricted+Sampling+Unrestricted+Sampling+Preserve+Label+Soft+Label+5+5+%2A+%2A+SST2+IMDb+SST2+IMDb+RNN+CNN+RNN+RNN+CNN+CNN+60%25+40%25+10%25+20%25+60%25+30%25+Unrestricted+Sampling+Unre) *Updated 24 October 2026*