--- title: "Entropy-aware Masking for Masked Language Modeling (2026)" description: "Standard masked language modeling chooses which tokens to hide uniformly at random. 13 references." authors: ["Gokul Srinivasagan", "Kai Hartung", "Munir Georges"] year: 2026 venue: "Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)" source: "https://aclanthology.org/2026.starsem-conference.27/" doi: "10.18653/v1/2026.starsem-conference.27" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/srinivasagan-2026-entropy-aware-masking-masked-language/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/srinivasagan-2026-entropy-aware-masking-masked-language/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) / Entropy-aware Masking for Masked Language Modeling # Entropy-aware Masking for Masked Language Modeling **Authors:** Gokul Srinivasagan, Kai Hartung and Munir Georges · **Year:** 2026 · **Venue:** [Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (\*SEM 2026)](https://md.research.elamur.ai/free/en/venues/starsem/) ([\*SEM 2026](https://md.research.elamur.ai/free/en/venues/starsem/2026/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/2026.starsem-conference.27/) · [PDF](https://aclanthology.org/2026.starsem-conference.27.pdf) · [DOI 10.18653/v1/2026.starsem-conference.27](https://doi.org/10.18653/v1/2026.starsem-conference.27) · [OpenAlex](https://openalex.org/W7166800409) **Citations:** 0 citations (OpenAlex, 2026-10-14) ## Summary Standard masked language modeling chooses which tokens to hide uniformly at random. This can waste training signal on tokens the model already predicts well. This short *SEM 2026 paper instead selects masks by entropy: the uncertainty of a model's predictive distribution at each position decides which 15% of tokens get masked. Entropy comes from one of two sources. In teacher-masking, it is computed by a fixed pretrained reference model. In self-masking, it is computed by the model under training, after a teacher-masking warm-up that avoids a cold start. The authors compare five selection rules by pretraining a 12-layer BERT and a 4-layer BERTlet on wikitext-103 with the MLM objective only, then fine-tuning on seven GLUE tasks. Masking the most uncertain tokens works best. Warm-started self-masking lifts the GLUE average by about 3–4 points over random masking, and combining it with knowledge distillation gives the strongest result. The method is cheap to adopt and especially helps small encoders. However, it has only been tested on English data and models up to about 110M parameters. ## Key points - **Models and setup:** BERT (109M parameters) and BERTlet (28.8M parameters) were pretrained on wikitext-103 for 25 epochs and evaluated on CoLA, MRPC, QNLI, QQP, RTE, SST-2 and MNLI. Hugging Face BERT checkpoints served as reference models. - **Teacher-masking:** High-entropy selection raised the GLUE average from 73.27 to 75.02 for BERT and from 70.93 to 74.07 for BERTlet. Mid-entropy selection was harmful, dropping BERT to 57.56. - **Self-masking:** - Without a teacher warm-up, it collapsed (BERT: 57.41). - With the warm-up, it reached 76.47 for BERT and 75.15 for BERTlet. - With a frozen encoder, the BERTlet gain was still about 3%. - **Scaling and distillation:** - Pretraining BERTlet on bookcorpus gave 77.09, compared with 71.66 for a randomly masked baseline. - Adding knowledge distillation only after the warm-up scored 77.19, versus 76.04 when distillation was applied from the start. - **Masking ratio:** Raising the ratio to 40% brought no clear gain. Masking a single token per sequence cost about 1 GLUE point. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* ## Institutions Also on the byline: [Technische Hochschule Ingolstadt](https://ror.org/02bxzcy64). 📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works 🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 1,344 works ## Cite Cite as: Gokul Srinivasagan, Kai Hartung and Munir Georges. 2026. Entropy-aware Masking for Masked Language Modeling. Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (\*SEM 2026). The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{srinivasagan2026entropy, title = {{Entropy-aware Masking for Masked Language Modeling}}, author = {Gokul Srinivasagan, Kai Hartung and Munir Georges}, booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)}, year = {2026}, doi = {10.18653/v1/2026.starsem-conference.27}, url = {https://doi.org/10.18653/v1/2026.starsem-conference.27} } ``` - [← previous Entity-Level Sentiment Analysis with Sentence Relevance Detection](https://md.research.elamur.ai/free/en/papers/ronningstad-2026-entity-level-sentiment-analysis-sentence/) - [next → EpiGator: An Event-based Surveillance System for Infectious Disease Outbreaks](https://md.research.elamur.ai/free/en/papers/wu-2026-epigator-event-based-surveillance-system/) - [↑ back to papers of 2026](https://md.research.elamur.ai/free/en/papers/2026/page/6/) ## References 13 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… 2. Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2020. Train no evil: Selective masking for task-guided pre-training. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6966–6974, Online. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Yuxian+Gu%2C+Zhengyan+Zhang%2C+Xiaozhi+Wang%2C+Zhiyuan+Liu%2C+and+Maosong+Sun%2E+2020%2E+Train+no+evil%3A+Selective+masking+for+task%2Dguided+pre%2Dtraining%2E+In+Proceedings+of+the+2020+Conference+on+Empirical+Methods+i) 3. Tanish Lad, Himanshu Maheshwari, Shreyas Kottukkal, and Radhika Mamidi. 2022. Using selective masking as a bridge between pre-training and fine-tuning. arXiv preprint arXiv:2211.13815. — [arXiv:2211.13815](https://arxiv.org/abs/2211.13815) [find](https://scholar.google.com/scholar?q=Tanish+Lad%2C+Himanshu+Maheshwari%2C+Shreyas+Kottukkal%2C+and+Radhika+Mamidi%2E+2022%2E+Using+selective+masking+as+a+bridge+between+pre%2Dtraining+and+fine%2Dtuning%2E+arXiv+preprint+arXiv%3A2211%2E13815%2E+%E2%80%94+%5BarXiv%3A2211%2E1) 4. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Zhenzhong+Lan%2C+Mingda+Chen%2C+Sebastian+Goodman%2C+Kevin+Gimpel%2C+Piyush+Sharma%2C+and+Radu+Soricut%2E+2020%2E+Albert%3A+A+lite+bert+for+self%2Dsupervised+learning+of+language+representations%2E+In+International+Confe) 5. Quentin Lhoest, Albert Villanova Del Moral, Yacine Jernite, Abhishek Thakur, Patrick Von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, and 1 others. 2021. [Datasets: A community library for natural language processing](https://md.research.elamur.ai/free/en/papers/lhoest-2021-datasets-community-library-natural-language/). In Proceedings of the 2021 conference on empiri… 6. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+Roberta%3A+A+robustly+optimized+bert+pretraining+approach%2E+a) 7. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/). arXiv preprint arXiv:1609.07843. 8. Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. [A primer in BERTology: What we know about how BERT works](https://md.research.elamur.ai/free/en/papers/rogers-2020-primer-bertology-what-we-know/). Transactions of the Association for Computational Linguistics, 8:842–866. 9. Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. [MobileBERT: a compact task-agnostic BERT for resource-limited devices](https://md.research.elamur.ai/free/en/papers/sun-2020-mobilebert-compact-task-agnostic-bert/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2158–2170, Online. Association for Computatio… 10. Alexander Wettig, Tianyu Gao, Zexuan Zhong, and Danqi Chen. 2023. Should you mask 15% in masked language modeling? In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2985– 3000, Dubrovnik, Croatia. Association for Computational Lingu… [find](https://scholar.google.com/scholar?q=Alexander+Wettig%2C+Tianyu+Gao%2C+Zexuan+Zhong%2C+and+Danqi+Chen%2E+2023%2E+Should+you+mask+15%25+in+masked+language+modeling%3F+In+Proceedings+of+the+17th+Conference+of+the+European+Chapter+of+the+Association+for+) 11. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. [Transf…](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/) 12. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, and 1 others. 2023. [A survey of large language models](https://md.research.elamur.ai/free/en/papers/zhao-2023-survey-large-language-models/). arXiv preprint arXiv:2303.18223, 1(2):1–124. 13. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. [Aligning books and movies: Towards story-like visual explanations by watching movies and reading books](https://md.research.elamur.ai/free/en/papers/zhu-2015-aligning-books-movies-towards-story/). In Proceedings of the IEEE international conference on computer vision, pages 19–… *Updated 24 October 2026*