--- title: "SenseBERT: Driving Some Sense into BERT — Levine 2020" description: "Self-supervised language models such as BERT learn from word forms, which are only an ambiguous stand-in for meaning." authors: ["Yoav Levine", "Barak Lenz", "Or Dagan", "Ori Ram", "Dan Padnos", "Or Sharir", "Shai Shalev-Shwartz", "Amnon Shashua", "Yoav Shoham"] year: 2020 venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/2020.acl-main.423/" doi: "10.18653/v1/2020.acl-main.423" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/levine-2020-sensebert-driving-some-sense-into/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/levine-2020-sensebert-driving-some-sense-into/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) / SenseBERT: Driving Some Sense into BERT # SenseBERT: Driving Some Sense into BERT **Authors:** Yoav Levine, Barak Lenz, Or Dagan, [Ori Ram](https://md.research.elamur.ai/free/en/authors/ori-ram/), Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, Yoav Shoham · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.423/) · [PDF](https://aclanthology.org/2020.acl-main.423.pdf) · [DOI 10.18653/v1/2020.acl-main.423](https://doi.org/10.18653/v1/2020.acl-main.423) · [OpenAlex](https://openalex.org/W2968289784) **Citations:** 162 citations (openalex-snapshot · 2026-10-14) · [96](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary Self-supervised language models such as BERT learn from word forms, which are only an ambiguous stand-in for meaning. For example, "bass" may denote a fish or an instrument. Levine et al. introduce SenseBERT, which adds a second pre-training objective: besides recovering each masked token, the model must predict its WordNet supersense. No sense-annotated corpus is required. WordNet lists the coarse categories each vocabulary item can take, and when a word has several candidates, a soft-labeling loss rewards probability placed on any of them. A supersense mapping is learned alongside BERT's word embedding matrix and also contributes to the input embeddings. The authors evaluate on SemEval-SS, a supersense version of the Raganato et al. (2017) WSD suite. SenseBERT clearly outperforms BERT there, with the largest gains when only a linear classifier is trained on frozen embeddings. A single SenseBERT LARGE model also set a new state of the art on WiC. The work shows that weak supervision from an expert-built lexical resource can make pre-trained representations more meaning-aware, while overall GLUE performance stays roughly unchanged. ## Key points - **Supersense inventory:** the model uses 45 WordNet supersense categories (26 noun, 15 verb, 3 adjective, 1 adverb). Short words, stop words and sub-word pieces receive no supersense label. - **SemEval-SS setup:** models are trained on SemCor (226,036 examples) and tested on SenseEval/SemEval data (7,253 examples). - Frozen setting: SenseBERT BASE scores 75.6 vs. 65.1 for BERT BASE, and SenseBERT LARGE scores 79.5 vs. 67.3 for BERT LARGE. - Fine-tuned setting: SenseBERT BASE reaches 83.0, above BERT LARGE at 81.1. - **WiC (SuperGLUE):** SenseBERT LARGE scores 72.1, about 2.5 points above BERT LARGE (69.6). It also beats RoBERTa (69.9), KnowBERT-W+W (70.9) and the BERT sense-embedding approach (67.7). - **GLUE:** the average score is essentially unchanged, at 77.9 for SenseBERT BASE vs. 77.5 for the authors' own BERT BASE. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 96 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 91 not yet. - [Annotating the French Wiktionary with supersenses for large scale lexical analysis: a use case to assess form-meaning relationships within the nominal lexicon](https://md.research.elamur.ai/free/en/papers/angleraud-2025-french-wiktionary-supersenses/) - [Entities as Experts: Sparse Memory Access with Entity Supervision](https://md.research.elamur.ai/free/en/papers/fevry-2020-entities-as-experts-sparse-memory/) - [Knowledge Enhanced Masked Language Model for Stance Detection](https://md.research.elamur.ai/free/en/papers/kawintiranon-2021-knowledge-enhanced-masked-language-model/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters](https://md.research.elamur.ai/free/en/papers/wang-2020-k-adapter-infusing-knowledge-pre/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{levine2020sensebert, title = {{SenseBERT: Driving Some Sense into BERT}}, author = {Levine, Yoav and Lenz, Barak and Dagan, Or and Ram, Ori and Padnos, Dan and Sharir, Or and Shalev-Shwartz, Shai and Shashua, Amnon and Shoham, Yoav}, booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics}, year = {2020}, doi = {10.18653/v1/2020.acl-main.423}, url = {https://doi.org/10.18653/v1/2020.acl-main.423} } ``` - [← previous Semi-supervised Contextual Historical Text Normalization](https://md.research.elamur.ai/free/en/papers/acl-2020-makarov-semi-supervised-contextual-historical-text/) - [next → Sentence Boundary Detection on Line Breaks in Japanese](https://md.research.elamur.ai/free/en/papers/aclbulk-2020-hayashibe-sentence-boundary-detection-line-breaks/) - [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/5/) ## References 1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. 5.4 GLUE The General Language Understanding Evaluation (GLUE; Wang et al. (2018)) benchmark is a popular testbed for language understanding models. It consists of 9 different NLP tasks, covering different linguistic phenomena. We evaluate our model on GLUE, in order to verify that SenseBERT gains it… [find](https://scholar.google.com/scholar?q=5%2E4+GLUE+The+General+Language+Understanding+Evaluation+%28GLUE%3B+Wang+et+al%2E+%282018%29%29+benchmark+is+a+popular+testbed+for+language+understanding+models%2E+It+consists+of+9+different+NLP+tasks%2C+covering+diffe)