--- title: "Supervised Contrastive Learning for Pre-trained — Gunel 2020" description: "Fine-tuning a pre-trained language model with cross-entropy alone tends to generalize poorly when labels are scarce, reacts badly to noisy labels, and…" authors: ["Beliz Gunel", "Jingfei Du", "Alexis Conneau", "Ves Stoyanov"] year: 2020 venue: "arXiv:2011.01403" source: "https://arxiv.org/abs/2011.01403" doi: "10.48550/arxiv.2011.01403" arxiv: "2011.01403" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/gunel-2020-supervised-contrastive-learning-pre-trained/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gunel-2020-supervised-contrastive-learning-pre-trained/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning # Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning **Authors:** Beliz Gunel, [Jingfei Du](https://md.research.elamur.ai/free/en/authors/jingfei-du/), [Alexis Conneau](https://md.research.elamur.ai/free/en/authors/alexis-conneau/), [Ves Stoyanov](https://md.research.elamur.ai/free/en/authors/ves-stoyanov/) · **Year:** 2020 · **Venue:** arXiv:2011.01403 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/2011.01403) · [PDF](https://arxiv.org/pdf/2011.01403) · [DOI 10.48550/arxiv.2011.01403](https://doi.org/10.48550/arxiv.2011.01403) · [arXiv 2011.01403](https://arxiv.org/abs/2011.01403) · [OpenAlex](https://openalex.org/W3096565276) **Citations:** 55 citations (openalex-snapshot · 2026-10-14) · [161](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Fine-tuning a pre-trained language model with cross-entropy alone tends to generalize poorly when labels are scarce, reacts badly to noisy labels, and produces results that vary a lot between runs. Gunel et al. address this by adding a supervised contrastive learning (SCL) term to the fine-tuning loss. Within each batch, the l2-normalized [CLS] embeddings of examples that share a label are pulled together, and those with different labels are pushed apart. A temperature τ controls how much hard negatives count, and a weight λ balances SCL against cross-entropy. Unlike self-supervised contrastive approaches, the method needs no augmented views, memory banks, extra unlabeled text or architectural changes. The authors fine-tune RoBERTa-Large on GLUE tasks (SST-2, CoLA, MRPC, RTE, QNLI, MNLI) with 20, 100 or 1000 labeled examples. They also test on noisy training sets built with back-translation and on transfer from SST-2 to related sentiment data. Improvements are largest in the smallest-data settings and shrink with full training sets. This makes SCL a cheap, drop-in option for low-resource text classification. ## Key points - **Few-shot gains:** SCL improves on the CE-only RoBERTa-Large baseline by up to 10.7 points with 20 labeled examples on SST-2, QNLI and MNLI. For example, QNLI rises from 65.0 to 75.7 and MNLI from 39.3 to 42.7 at N=20. - **Robustness to noise:** The authors build noisy training sets with a back-translation model, using its sampling temperature to set the noise level. On these sets, CE+SCL beats CE by up to 7 points. - **Transfer:** A model fine-tuned on SST-2 and transferred to Amazon-2 gains 2.9 points over the CE-only model. It also shows much lower variance across few-shot training samples. - **Full-data results and ablations:** With full training data, gains are modest and mainly significant on MRPC (86.4 → 89.5) and QNLI (90.4 → 93.9). A "CE+CE" ablation keeps the normalization and temperature but drops SCL, and it recovers much of the improvement. The paper also includes a comparison with Khosla et al.'s (2020) approach, which trains in two separate stages. - **Cost and batch size (RoBERTa-Base):** SCL slows fine-tuning somewhat, for example 1.54 vs 2.46 updates/sec at batch size 256. Larger batches help it slightly: MNLI improves from 85.3 at batch size 16 to 86.6 at 256. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Other](https://md.research.elamur.ai/free/en/shelves/other/) ## Cited by Cited by 161 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 151 not yet. - [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/cao-2021-cliff-contrastive-learning-improving-faithfulness/) - [TATA: Stance Detection via Topic-Agnostic and Topic-Aware Embeddings](https://md.research.elamur.ai/free/en/papers/hanley-2023-tata-stance-detection-topic-agnostic/) - [Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition](https://md.research.elamur.ai/free/en/papers/jiang-2023-global-local-hierarchy-aware-contrastive/) - [Self-Guided Contrastive Learning for BERT Sentence Representations](https://md.research.elamur.ai/free/en/papers/kim-2021-self-guided-contrastive-learning-bert/) - [JointCL: A Joint Contrastive Learning Framework for Zero-Shot Stance Detection](https://md.research.elamur.ai/free/en/papers/liang-2022-jointcl-joint-contrastive-learning-framework/) - [Does DetectGPT Fully Utilize Perturbation? Bridging Selective Perturbation to Fine-tuned Contrastive Learning Detector would be Better](https://md.research.elamur.ai/free/en/papers/liu-2024-does-detectgpt-fully-utilize-perturbation/) - [Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations](https://md.research.elamur.ai/free/en/papers/long-2022-facilitating-contrastive-learning-discourse-relational/) - [COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining](https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/) - [HCMUS_The Fangs at AbjadStyleTransfer Shared Task: Learning to Query Style, Contrastive Representations for Zero-Shot Arabic Authorship Style Transfer](https://md.research.elamur.ai/free/en/papers/sy-2026-hcmus-fangs-abjadstyletransfer-shared-task/) - [Supervised Contrastive Fine-Tuning for Active Few-Shot Learning](https://md.research.elamur.ai/free/en/papers/zhang-2026-supervised-contrastive-fine-tuning-active/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{gunel2020supervised, title = {{Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning}}, author = {Gunel, Beliz and Du, Jingfei and Conneau, Alexis and Stoyanov, Ves}, year = {2020}, doi = {10.48550/arxiv.2011.01403}, eprint = {2011.01403}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.2011.01403} } ``` - [← previous Studies in Semitic Vocalisation and Reading Traditions](https://md.research.elamur.ai/free/en/papers/hbobook-obp0207-hornkohl-studies-semitic-vocalisation-reading-traditions/) - [next → Syntactic Data Augmentation Increases Robustness to Inference Heuristics](https://md.research.elamur.ai/free/en/papers/min-2020-syntactic-data-augmentation-increases-robustness/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) ## References 2 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. Better fine-tuning by reducing representational collapse. ArXiv, abs/2008.03156, 2020. — [arXiv:2008.03156](https://arxiv.org/abs/2008.03156) [find](https://scholar.google.com/scholar?q=Armen+Aghajanyan%2C+Akshat+Shrivastava%2C+Anchit+Gupta%2C+Naman+Goyal%2C+Luke+Zettlemoyer%2C+and+Sonal+Gupta%2E+Better+fine%2Dtuning+by+reducing+representational+collapse%2E+ArXiv%2C+abs%2F2008%2E03156%2C+2020%2E+%E2%80%94+%5BarXiv%3A2008) 2. MRPC RTE QNLI MNLI Avg 86.4±2.4 89.5±0.9 85.5±1.8 85.7±0.5 90.4±0.8 93.9±0.7 88.4±1 88.6±0.7 88.8 90 0.01 0.06 0.01 0.16 CoLA QNLI MNLI Avg ups/sec 83.3±0.7 83.7±0.9 88.2±0.8 92.5±0.4 84±0.6 85.3±0.5 15.9 15.08 83.3±0.5 83.8±0.6 89.2±0.5 92.6±0.5 84±0.4 85.7±0.7 8.43 7.44 2.46 1.54 84±0.5 84.5±0.5 9… [find](https://scholar.google.com/scholar?q=MRPC+RTE+QNLI+MNLI+Avg+86%2E4%C2%B12%2E4+89%2E5%C2%B10%2E9+85%2E5%C2%B11%2E8+85%2E7%C2%B10%2E5+90%2E4%C2%B10%2E8+93%2E9%C2%B10%2E7+88%2E4%C2%B11+88%2E6%C2%B10%2E7+88%2E8+90+0%2E01+0%2E06+0%2E01+0%2E16+CoLA+QNLI+MNLI+Avg+ups%2Fsec+83%2E3%C2%B10%2E7+83%2E7%C2%B10%2E9+88%2E2%C2%B10%2E8+92%2E5%C2%B10%2E4+84%C2%B10%2E6+85%2E3%C2%B10%2E5+)