---
title: "A Better Way to Do Masked Language Model Scoring — Kauf 2023"
description: "They propose PLL-word-l2r. 22 references, cited by 34 papers here. Among them: 19."
authors: ["Carina Kauf", "Anna A. Ivanova"]
year: 2023
venue: "Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)"
source: "https://aclanthology.org/2023.acl-short.80/"
doi: "10.18653/v1/2023.acl-short.80"
arxiv: "2305.10588"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kauf-2023-better-way-do-masked-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kauf-2023-better-way-do-masked-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/) / A Better Way to Do Masked Language Model Scoring

# A Better Way to Do Masked Language Model Scoring

**Authors:** [Carina Kauf](https://md.research.elamur.ai/free/en/authors/carina-kauf/) and [Anna A. Ivanova](https://md.research.elamur.ai/free/en/authors/anna-a-ivanova/) · **Year:** 2023 · **Venue:** [Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2023](https://md.research.elamur.ai/free/en/venues/acl/2023/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2023.acl-short.80/) · [PDF](https://aclanthology.org/2023.acl-short.80.pdf) · [DOI 10.18653/v1/2023.acl-short.80](https://doi.org/10.18653/v1/2023.acl-short.80) · [arXiv 2305.10588](https://arxiv.org/abs/2305.10588) · [OpenAlex](https://openalex.org/W4385893893)

**Citations:** 19 citations (OpenAlex, 2026-10-14) · [34](https://md.research.elamur.ai/free/en/papers/kauf-2023-better-way-do-masked-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Masked language models such as BERT have no natural way to assign a probability to a whole sentence. Researchers therefore commonly use the pseudo-log-likelihood (PLL) of Salazar et al. (2020), which masks each token in turn and adds up the log probabilities. Kauf and Ivanova show that this standard variant overrates words that the tokenizer splits into several subword pieces, because each piece is easy to guess when the other pieces of the same word stay visible.

They propose PLL-word-l2r. When it scores a subword token, it also hides the later pieces of that same word. They compare it with the original metric and with PLL-whole-word, which hides the entire word. The experiments use an adapted minicons scorer, bert-base-cased with gpt2-medium as a reference, and the EventsAdapt sentences, with extra checks on Brown and LibriSpeech.

PLL-word-l2r best meets the expected properties. Longer sentences receive lower scores, frequent words receive higher scores, and agreement with GPT-2 log-likelihoods is strongest. The choice of metric even changes BLiMP results, so it matters for any MLM evaluation.

## Key points
- **Main dataset:** EventsAdapt has 782 sentence pairs of 5–7 words each. It is rich in out-of-vocabulary words, which make up 19.6% of words for BERT and 40.3% for GPT-2. Results also generalize to the Brown corpus and LibriSpeech, though the effects are weaker there because these corpora contain fewer OOV words.
- **Problems with PLL-original:** It breaks expected trends. Its scores show the wrong relation to sentence length, correlate weakly with word frequency and correlate poorly with gpt2-medium LL scores.
- **PLL-word-l2r vs. PLL-whole-word:** PLL-word-l2r slightly beats PLL-whole-word, which seems to penalize multi-token words too harshly.
- **BLiMP (67k minimal pairs):** PLL-word-l2r gives the top overall score for all four tested models:
  - BERT-base: 84.7 vs. 84.2 (original)
  - BERT-large: 85.0 vs. 84.8
  - RoBERTa-base: 86.7 vs. 85.4
  - RoBERTa-large: 87.5 vs. 86.5
- **Limitations and availability:** The method was tested only on English, and the authors note it may behave differently for agglutinative languages, which is relevant for morphologically rich languages such as Ukrainian. Code is available at github.com/carina-kauf/better-mlm-scoring.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Massachusetts Institute of Technology](https://md.research.elamur.ai/free/en/universities/massachusetts-institute-of-technology/) — University, Cambridge, United States · 243 works · 76,506 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Carina Kauf and Anna A. Ivanova. 2023. A Better Way to Do Masked Language Model Scoring. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kauf2023better,
  title = {{A Better Way to Do Masked Language Model Scoring}},
  author = {Carina Kauf and Anna A. Ivanova},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.80},
  eprint = {2305.10588},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-short.80}
}
```

- [← previous SLM: Bridge the thin gap between speech and text foundation models](https://md.research.elamur.ai/free/en/papers/wang-2023-slm-bridge-thin-gap-between/)
- [next → DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence](https://md.research.elamur.ai/free/en/papers/zhao-2023-discoscore-evaluating-text-generation-bert/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/10/)

## References

22 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sudeep Bhatia and Russell Richie. 2022. Transformer networks of human conceptual knowledge. Psychological Review. [find](https://scholar.google.com/scholar?q=Sudeep+Bhatia+and+Russell+Richie%2E+2022%2E+Transformer+networks+of+human+conceptual+knowledge%2E+Psychological+Review%2E)
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
3. Evelina Fedorenko, Idan Asher Blank, Matthew Siegelman, and Zachary Mineroff. 2020. Lack of selectivity for syntax relative to word meanings throughout the language network. Cognition, 203:104348. [find](https://scholar.google.com/scholar?q=Evelina+Fedorenko%2C+Idan+Asher+Blank%2C+Matthew+Siegelman%2C+and+Zachary+Mineroff%2E+2020%2E+Lack+of+selectivity+for+syntax+relative+to+word+meanings+throughout+the+language+network%2E+Cognition%2C+203%3A104348%2E)
4. W Nelson Francis and Henry Kucera. 1979. Brown corpus manual. Letters to the Editor, 5(2):7. [find](https://scholar.google.com/scholar?q=W+Nelson+Francis+and+Henry+Kucera%2E+1979%2E+Brown+corpus+manual%2E+Letters+to+the+Editor%2C+5%282%29%3A7%2E)
5. Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. [Syntaxgym: An online platform for targeted evaluation of language models](https://md.research.elamur.ai/free/en/papers/gauthier-2020-syntaxgym-online-platform-targeted-evaluation/). Association for Computational Linguistics (ACL).
6. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. [Surface form competition: Why the highest probability answer isn’t always right](https://md.research.elamur.ai/free/en/papers/holtzman-2021-surface-form-competition-why-highest/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7038–7051, Online and Punta Cana, Dominic…
7. Carina Kauf, Anna A Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan S She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2022. Event knowledge in large language models: the gap between the impossible and the unlikely. arXiv preprint arXiv:2212.01488. [find](https://scholar.google.com/scholar?q=Carina+Kauf%2C+Anna+A+Ivanova%2C+Giulia+Rambelli%2C+Emmanuele+Chersoni%2C+Jingyuan+S+She%2C+Zawad+Chowdhury%2C+Evelina+Fedorenko%2C+and+Alessandro+Lenci%2E+2022%2E+Event+knowledge+in+large+language+models%3A+the+gap+betw)
8. [Masked language modeling and the distributional hypothesis: Order word matters pre-training for little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2888–2913, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
9. Philippe Laban, Tobias Schnabel, Paul Bennett, and Marti A. Hearst. 2021. Keep it simple: Unsupervised simplification of multi-paragraph text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pr… [find](https://scholar.google.com/scholar?q=Philippe+Laban%2C+Tobias+Schnabel%2C+Paul+Bennett%2C+and+Marti+A%2E+Hearst%2E+2021%2E+Keep+it+simple%3A+Unsupervised+simplification+of+multi%2Dparagraph+text%2E+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Associat)
10. Alex Wang and Kyunghyun Cho. 2019. [BERT has a mouth, and it must speak: BERT as a Markov random field language model](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/). In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30–36, Minneapolis, Minnesota. Association for Computational Linguistics.
11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692.
12. Kanishka Misra. 2022. [minicons: Enabling flexible behavioral and representational analyses of transformer language models](https://md.research.elamur.ai/free/en/papers/misra-2022-minicons-enabling-flexible-behavioral-representational/). arXiv preprint arXiv:2203.13112.
13. Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020. [CrowS-pairs: A challenge dataset for measuring social biases in masked language models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1953–1967, Online. Association for…
14. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+2015%2E+Librispeech%3A+an+asr+corpus+based+on+public+domain+audio+books%2E+In+2015+IEEE+international+conference+on+acoustics%2C+speech+and+)
15. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+Ilya+Sutskever%2C+et+al%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+OpenAI+blog%2C+1%288%29%3A9%2E)
16. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. [BLiMP: The benchmark of linguistic minimal pairs for english](https://md.research.elamur.ai/free/en/papers/warstadt-2020-blimp-benchmark-linguistic-minimal-pairs/). Transactions of the Association for Computational Linguistics, 8:377– 392.
17. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, et al. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P…
18. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel R. Bowman. 2021. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1… [find](https://scholar.google.com/scholar?q=Yian+Zhang%2C+Alex+Warstadt%2C+Xiaocheng+Li%2C+and+Samuel+R%2E+Bowman%2E+2021%2E+When+do+you+need+billions+of+words+of+pretraining+data%3F+In+Proceedings+of+the+59th+Annual+Meeting+of+the+Association+for+Computatio)
19. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2020. [Masked language model scoring](https://md.research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712.
20. Joonbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective sentence scoring method using bert for speech recognition. In Asian Conference on Machine Learning, pages 1081–1093. PMLR. [find](https://scholar.google.com/scholar?q=Joonbo+Shin%2C+Yoonhyung+Lee%2C+and+Kyomin+Jung%2E+2019%2E+Effective+sentence+scoring+method+using+bert+for+speech+recognition%2E+In+Asian+Conference+on+Machine+Learning%2C+pages+1081%E2%80%931093%2E+PMLR%2E)
21. Arabella Sinclair, Jaap Jumelet, Willem Zuidema, and Raquel Fernández. 2022. [Structural persistence in language models: Priming as a window into abstract language representations](https://md.research.elamur.ai/free/en/papers/sinclair-2022-structural-persistence-language-models-priming/). Transactions of the Association for Computational Linguistics, 10:1031–1050.
22. Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. 2021. *Figure 5: The PLL-original metric inflates the score of the word carnivore. PLL-word-l2r mitigate this issue, whereas PLL-whole-word overly penalizes the word. Model: bert-base-cased.* *Figure 6: The PLL… [find](https://scholar.google.com/scholar?q=Koustuv+Sinha%2C+Robin+Jia%2C+Dieuwke+Hupkes%2C+Joelle+Pineau%2C+Adina+Williams%2C+and+Douwe+Kiela%2E+2021%2E+%2AFigure+5%3A+The+PLL%2Doriginal+metric+inflates+the+score+of+the+word+carnivore%2E+PLL%2Dword%2Dl2r+mitigate+this+)

*Updated 2 November 2026*