---
title: "Masked Language Model Scoring — Salazar 2020"
description: "Masked language models such as BERT are usually finetuned before use, and they lack an obvious way to assign probabilities to whole sentences."
authors: ["Julian Salazar", "Davis Liang", "Toan Q. Nguyen", "Katrin Kirchhoff"]
year: 2020
venue: "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics"
source: "https://aclanthology.org/2020.acl-main.240/"
doi: "10.18653/v1/2020.acl-main.240"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/salazar-2020-masked-language-model-scoring/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/) / Masked Language Model Scoring

# Masked Language Model Scoring

**Authors:** Julian Salazar, [Davis Liang](https://md.research.elamur.ai/free/en/authors/davis-liang/), [Toan Q. Nguyen](https://md.research.elamur.ai/free/en/authors/toan-q-nguyen/), [Katrin Kirchhoff](https://md.research.elamur.ai/free/en/authors/katrin-kirchhoff/) · **Year:** 2020 · **Venue:** Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/2020.acl-main.240/) · [PDF](https://aclanthology.org/2020.acl-main.240.pdf) · [DOI 10.18653/v1/2020.acl-main.240](https://doi.org/10.18653/v1/2020.acl-main.240) · [OpenAlex](https://openalex.org/W3024308166)

**Citations:** 191 citations (openalex-snapshot · 2026-10-14) · [253](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary

Masked language models such as BERT are usually finetuned before use, and they lack an obvious way to assign probabilities to whole sentences. This makes them seem unsuitable for tasks that rank candidate outputs, unlike left-to-right models like GPT-2. Salazar et al. address this with pseudo-log-likelihood (PLL). Each token in a sentence is hidden in turn, the model predicts it from the surrounding context on both sides, and the log probabilities of these predictions are added together. The authors use PLLs from pretrained BERT, RoBERTa and XLM to rerank n-best hypotheses from end-to-end speech recognition (LibriSpeech) and neural translation (TED Talks, IWSLT 2015, WMT 2014). They also test grammaticality judgments on BLiMP without any supervision. In addition, they define pseudo-perplexity for evaluating a model on a corpus, and they show that a finetuned model can estimate PLL in one forward pass instead of one pass per token. The masked models match or beat GPT-2, which suggests that pretrained MLMs can be dropped into existing systems as sentence scorers.

## Key points

- **Speech recognition:** Rescoring with RoBERTa large lowers WER by about 30% relative on LibriSpeech test-clean and 18% on test-other. Domain adaptation brings further gains.
- **Translation:** Low-resource TED Talks pairs, such as Galician→English with roughly 10k sentence pairs, improve by up to +1.7 BLEU. A single 15-language XLM improves translations into several target languages at once.
- **Grammaticality (BLiMP):** BERT and RoBERTa beat GPT-2 by up to 3.9% absolute, with roughly +10 points on island effects and NPI licensing.
- **Faster scoring:** Regression on the [CLS] token lets a model approximate PLL without masking. This replaces the cost of one pass per token with a single pass.
- **Code:** The scoring library is released at github.com/awslabs/mlm-scoring.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

253 papers of our library cite it (of the 140673 whose bibliographies we read); 28 of them have a page here, 225 not yet.

- [We Understand Elliptical Sentences, and Language Models should Too: A New Dataset for Studying Ellipsis and its Interaction with Thematic Fit](https://md.research.elamur.ai/free/en/papers/acl-2023-testa-we-understand-elliptical-sentences-language/)
- [Systematic Textual Availability of Manuscripts](https://md.research.elamur.ai/free/en/papers/aclbulk-2025-miller-systematic-textual-availability-manuscripts/)
- [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://md.research.elamur.ai/free/en/papers/agarwal-2025-mechanisms-vs-outcomes-probing-syntax/)
- [The User-Aware Arabic Gender Rewriter](https://md.research.elamur.ai/free/en/papers/alhafni-2023-user-aware-arabic-gender-rewriter/)
- [Probing Pre-Trained Language Models for Cross-Cultural Differences in Values](https://md.research.elamur.ai/free/en/papers/arora-2023-probing-pre-trained-language-models/)
- [Multilingual Nonce Dependency Treebanks: Understanding how Language Models Represent and Process Syntactic Structure](https://md.research.elamur.ai/free/en/papers/arps-2024-multilingual-nonce-dependency-treebanks-understanding/)
- [Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer](https://md.research.elamur.ai/free/en/papers/briakou-2021-evaluating-evaluation-metrics-style-transfer/)
- [Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution](https://md.research.elamur.ai/free/en/papers/emelin-2021-wino-x-multilingual-winograd-schemas/)
- [Bias and Fairness in Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/gallegos-2023-bias-fairness-large-language-models/)
- [Prompting is not a substitute for probability measurements in large language models](https://md.research.elamur.ai/free/en/papers/hu-2023-prompting-is-not-substitute-probability/)
- [LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion](https://md.research.elamur.ai/free/en/papers/jiang-2023-llm-blender-ensembling-large-language/)
- [POLITICS: Pretraining with Same-story Article Comparison for Ideology Prediction and Stance Detection](https://md.research.elamur.ai/free/en/papers/liu-2022-politics-pretraining-same-story-article/)
- [TimeLMs: Diachronic Language Models from Twitter](https://md.research.elamur.ai/free/en/papers/loureiro-2022-timelms-diachronic-language-models-twitter/)
- [LENS: A Learnable Evaluation Metric for Text Simplification](https://md.research.elamur.ai/free/en/papers/maddela-2023-lens-learnable-evaluation-metric-text/)
- [An Empirical Survey of the Effectiveness of Debiasing Techniques for Pre-trained Language Models](https://md.research.elamur.ai/free/en/papers/meade-2022-empirical-survey-effectiveness-debiasing-techniques/)
- [Towards Content Accessibility Through Lexical Simplification for Maltese as a Low-Resource Language](https://md.research.elamur.ai/free/en/papers/meli-2024-content-accessibility-through-lexical-simplification/)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/min-2021-recent-advances-natural-language-processing/)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://md.research.elamur.ai/free/en/papers/nadeem-2021-stereoset-measuring-stereotypical-bias-pretrained/)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://md.research.elamur.ai/free/en/papers/nangia-2020-crows-pairs-challenge-dataset-measuring/)
- [Having Beer after Prayer? Measuring Cultural Bias in Large Language Models](https://md.research.elamur.ai/free/en/papers/naous-2023-having-beer-after-prayer-measuring/)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://md.research.elamur.ai/free/en/papers/schick-2021-it-s-not-just-size/)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://md.research.elamur.ai/free/en/papers/schick-2021-self-diagnosis-self-debiasing-proposal/)
- [Language Models Learn Constructional Semantics, Not To Mention Syntax: Investigating LM Understanding of Paired-Focus Constructions](https://md.research.elamur.ai/free/en/papers/scivetti-2026-language-models-learn-constructional-semantics/)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://md.research.elamur.ai/free/en/papers/sinha-2021-masked-language-modeling-distributional-hypothesis/)
- [“I'm sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset](https://md.research.elamur.ai/free/en/papers/smith-2022-i-m-sorry-hear-that/)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
- [Contextual Probing for Low-Resource Named Entity Recognition in Latin](https://md.research.elamur.ai/free/en/papers/trusca-2026-contextual-probing-low-resource-named/)
- [The Mark Fades: Adaptive Evolutionary Paraphrase-based Attack against LLM Watermarks](https://md.research.elamur.ai/free/en/papers/zhao-2026-mark-fades-adaptive-evolutionary-paraphrase/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{salazar2020masked,
  title = {{Masked Language Model Scoring}},
  author = {Salazar, Julian and Liang, Davis and Nguyen, Toan Q. and Kirchhoff, Katrin},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.240},
  url = {https://doi.org/10.18653/v1/2020.acl-main.240}
}
```

- [← previous Many systems, one strategy: Acquiring ordinals in Dutch and English](https://md.research.elamur.ai/free/en/papers/meyer-2020-many-systems-one-strategy-acquiring/)
- [next → MASRI-HEADSET: A Maltese Corpus for Speech Recognition](https://md.research.elamur.ai/free/en/papers/mena-2020-masri-headset-maltese-corpus-speech/)
- [↑ back to papers of 2020](https://md.research.elamur.ai/free/en/papers/2020/page/4/)

## References

58 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Roee+Aharoni%2C+Melvin+Johnson%2C+and+Orhan+Firat%2E+2019%2E+Massively+multilingual+neural+machine+translation%2E+In+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/)
2. Ebru Arisoy, Abhinav Sethy, Bhuvana Ramabhadran, and Stanley Chen. 2015. Bidirectional recurrent neural network language models for automatic speech recognition. In ICASSP. [find](https://scholar.google.com/scholar?q=Ebru+Arisoy%2C+Abhinav+Sethy%2C+Bhuvana+Ramabhadran%2C+and+Stanley+Chen%2E+2015%2E+Bidirectional+recurrent+neural+network+language+models+for+automatic+speech+recognition%2E+In+ICASSP%2E)
3. Julian Besag. 1975. Statistical analysis of non-lattice data. The Statistician, 24(3):179–195. [find](https://scholar.google.com/scholar?q=Julian+Besag%2E+1975%2E+Statistical+analysis+of+non%2Dlattice+data%2E+The+Statistician%2C+24%283%29%3A179%E2%80%93195%2E)
4. Peter F Brown, Vincent J Della Pietra, Stephen A Della Pietra, and Robert L Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263–311. [find](https://scholar.google.com/scholar?q=Peter+F+Brown%2C+Vincent+J+Della+Pietra%2C+Stephen+A+Della+Pietra%2C+and+Robert+L+Mercer%2E+1993%2E+The+mathematics+of+statistical+machine+translation%3A+Parameter+estimation%2E+Computational+Linguistics%2C+19%282%29%3A263)
5. Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, R Cattoni, and Marcello Federico. 2015. [find](https://scholar.google.com/scholar?q=Mauro+Cettolo%2C+Jan+Niehues%2C+Sebastian+Stu%CC%88ker%2C+Luisa+Bentivogli%2C+R+Cattoni%2C+and+Marcello+Federico%2E+2015%2E)
6. The IWSLT 2015 evaluation campaign. Technical report, FBK and KIT. [find](https://scholar.google.com/scholar?q=The+IWSLT+2015+evaluation+campaign%2E+Technical+report%2C+FBK+and+KIT%2E)
7. William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. 2016. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In ICASSP. [find](https://scholar.google.com/scholar?q=William+Chan%2C+Navdeep+Jaitly%2C+Quoc+Le%2C+and+Oriol+Vinyals%2E+2016%2E+Listen%2C+attend+and+spell%3A+A+neural+network+for+large+vocabulary+conversational+speech+recognition%2E+In+ICASSP%2E)
8. Xie Chen, Anton Ragni, Xunying Liu, and Mark JF Gales. 2017. Investigating bidirectional recurrent neural network language models for speech recognition. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Xie+Chen%2C+Anton+Ragni%2C+Xunying+Liu%2C+and+Mark+JF+Gales%2E+2017%2E+Investigating+bidirectional+recurrent+neural+network+language+models+for+speech+recognition%2E+In+INTERSPEECH%2E)
9. Noam Chomsky. 1957. Syntactic structures. Mouton. [find](https://scholar.google.com/scholar?q=Noam+Chomsky%2E+1957%2E+Syntactic+structures%2E+Mouton%2E)
10. Stephane Clinchant, Kweon Woo Jung, and Vassilina Nikoulina. 2019. On the use of BERT for neural machine translation. In WNGT. [find](https://scholar.google.com/scholar?q=Stephane+Clinchant%2C+Kweon+Woo+Jung%2C+and+Vassilina+Nikoulina%2E+2019%2E+On+the+use+of+BERT+for+neural+machine+translation%2E+In+WNGT%2E)
11. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. Deep Learning Workshop, NeurIPS. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2C+Oriol+Vinyals%2C+and+Jeff+Dean%2E+2014%2E+Distilling+the+knowledge+in+a+neural+network%2E+Deep+Learning+Workshop%2C+NeurIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/hinton-2015-distilling-knowledge-neural-network/)
12. Jiaji Huang, Yi Li, Wei Ping, and Liang Huang. 2018. Large margin neural language model. In EMNLP. [find](https://scholar.google.com/scholar?q=Jiaji+Huang%2C+Yi+Li%2C+Wei+Ping%2C+and+Liang+Huang%2E+2018%2E+Large+margin+neural+language+model%2E+In+EMNLP%2E)
13. Frederick Jelinek, Lalit Bahl, and Robert Mercer. 1975. Design of a linguistic statistical decoder for the recognition of continuous speech. IEEE Trans. Inf. Theory, 21(3):250–256. [find](https://scholar.google.com/scholar?q=Frederick+Jelinek%2C+Lalit+Bahl%2C+and+Robert+Mercer%2E+1975%2E+Design+of+a+linguistic+statistical+decoder+for+the+recognition+of+continuous+speech%2E+IEEE+Trans%2E+Inf%2E+Theory%2C+21%283%29%3A250%E2%80%93256%2E)
14. Yoon Kim and Alexander M Rush. 2016. Sequence-level knowledge distillation. In EMNLP. [find](https://scholar.google.com/scholar?q=Yoon+Kim+and+Alexander+M+Rush%2E+2016%2E+Sequence%2Dlevel+knowledge+distillation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2016-sequence-level-knowledge-distillation/)
15. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, 41(5):1202–1241. [find](https://scholar.google.com/scholar?q=Jey+Han+Lau%2C+Alexander+Clark%2C+and+Shalom+Lappin%2E+2017%2E+Grammaticality%2C+acceptability%2C+and+probability%3A+A+probabilistic+view+of+linguistic+knowledge%2E+Cognitive+Science%2C+41%285%29%3A1202%E2%80%931241%2E)
16. Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M Cohen, Huyen Nguyen, and Ravi Teja Gadde. 2019. Jasper: An end-to-end convolutional neural acoustic model. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Jason+Li%2C+Vitaly+Lavrukhin%2C+Boris+Ginsburg%2C+Ryan+Leary%2C+Oleksii+Kuchaiev%2C+Jonathan+M+Cohen%2C+Huyen+Nguyen%2C+and+Ravi+Teja+Gadde%2E+2019%2E+Jasper%3A+An+end%2Dto%2Dend+convolutional+neural+acoustic+model%2E+In+INTER)
17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. [find](https://scholar.google.com/scholar?q=Yinhan+Liu%2C+Myle+Ott%2C+Naman+Goyal%2C+Jingfei+Du%2C+Mandar+Joshi%2C+Danqi+Chen%2C+Omer+Levy%2C+Mike+Lewis%2C+Luke+Zettlemoyer%2C+and+Veselin+Stoyanov%2E+2019%2E+RoBERTa%3A+A+robustly+optimized+BERT+pretraining+approach%2E+a)
18. Kenton Murray and David Chiang. 2018. Correcting length bias in neural machine translation. In WMT. [find](https://scholar.google.com/scholar?q=Kenton+Murray+and+David+Chiang%2E+2018%2E+Correcting+length+bias+in+neural+machine+translation%2E+In+WMT%2E)
19. Graham Neubig and Junjie Hu. 2018. Rapid adaptation of neural machine translation to new languages. In EMNLP. [find](https://scholar.google.com/scholar?q=Graham+Neubig+and+Junjie+Hu%2E+2018%2E+Rapid+adaptation+of+neural+machine+translation+to+new+languages%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/neubig-2018-rapid-adaptation-neural-machine-translation/)
20. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In NeurIPS. [find](https://scholar.google.com/scholar?q=Alexis+Conneau+and+Guillaume+Lample%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+In+NeurIPS%2E)
21. Toan Q Nguyen and David Chiang. 2018. Improving lexical choice in neural machine translation. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Toan+Q+Nguyen+and+David+Chiang%2E+2018%2E+Improving+lexical+choice+in+neural+machine+translation%2E+In+NAACL%2DHLT%2E)
22. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In ACL. [find](https://scholar.google.com/scholar?q=Zihang+Dai%2C+Zhilin+Yang%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Quoc+V+Le%2C+and+Ruslan+Salakhutdinov%2E+2019%2E+Transformer%2DXL%3A+Attentive+language+models+beyond+a+fixed%2Dlength+context%2E+In+ACL%2E)
23. Toan Q Nguyen and Julian Salazar. 2019. Transformers without tears: Improving the normalization of self-attention. In IWSLT. [find](https://scholar.google.com/scholar?q=Toan+Q+Nguyen+and+Julian+Salazar%2E+2019%2E+Transformers+without+tears%3A+Improving+the+normalization+of+self%2Dattention%2E+In+IWSLT%2E)
24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+NAACL%2DHLT%2E)
25. Andrew Finch and Eiichiro Sumita. 2009. Bidirectional phrase-based statistical machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Andrew+Finch+and+Eiichiro+Sumita%2E+2009%2E+Bidirectional+phrase%2Dbased+statistical+machine+translation%2E+In+EMNLP%2E)
26. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage re-ranking with BERT. arXiv preprint arXiv:1901.04085. [find](https://scholar.google.com/scholar?q=Rodrigo+Nogueira+and+Kyunghyun+Cho%2E+2019%2E+Passage+re%2Dranking+with+BERT%2E+arXiv+preprint+arXiv%3A1901%2E04085%2E) [our card](https://md.research.elamur.ai/free/en/papers/nogueira-2019-passage-re-ranking-bert/)
27. Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis C Cobo, Florian Stimberg, et al. 2018. Parallel WaveNet: Fast high-fidelity speech synthesis. In ICML. [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Yazhe+Li%2C+Igor+Babuschkin%2C+Karen+Simonyan%2C+Oriol+Vinyals%2C+Koray+Kavukcuoglu%2C+George+van+den+Driessche%2C+Edward+Lockhart%2C+Luis+C+Cobo%2C+Florian+Stimberg%2C+et+al%2E+2018%2E+Parallel+WaveNet)
28. Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2015. On using monolingual corpora in neural machine translation. arXiv preprint arXiv:1503.03535. [find](https://scholar.google.com/scholar?q=Caglar+Gulcehre%2C+Orhan+Firat%2C+Kelvin+Xu%2C+Kyunghyun+Cho%2C+Loic+Barrault%2C+Huei%2DChi+Lin%2C+Fethi+Bougares%2C+Holger+Schwenk%2C+and+Yoshua+Bengio%2E+2015%2E+On+using+monolingual+corpora+in+neural+machine+translation) [our card](https://md.research.elamur.ai/free/en/papers/gulcehre-2015-using-monolingual-corpora-neural-machine/)
29. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In ICASSP. [find](https://scholar.google.com/scholar?q=Vassil+Panayotov%2C+Guoguo+Chen%2C+Daniel+Povey%2C+and+Sanjeev+Khudanpur%2E+2015%2E+LibriSpeech%3A+An+ASR+corpus+based+on+public+domain+audio+books%2E+In+ICASSP%2E)
30. Jian Guo, He He, Tong He, Leonard Lausen, Mu Li, Haibin Lin, Xingjian Shi, Chenguang Wang, Junyuan Xie, Sheng Zha, et al. 2020. GluonCV and GluonNLP: Deep learning in computer vision and natural language processing. JMLR, 21(23):1–7. [find](https://scholar.google.com/scholar?q=Jian+Guo%2C+He+He%2C+Tong+He%2C+Leonard+Lausen%2C+Mu+Li%2C+Haibin+Lin%2C+Xingjian+Shi%2C+Chenguang+Wang%2C+Junyuan+Xie%2C+Sheng+Zha%2C+et+al%2E+2020%2E+GluonCV+and+GluonNLP%3A+Deep+learning+in+computer+vision+and+natural+langu)
31. Ye Qi, Devendra Singh Sachan, Matthieu Felix, Sarguna Janani Padmanabhan, and Graham Neubig. 2018. When and why are pre-trained word embeddings useful for neural machine translation? In NAACL-HLT. [find](https://scholar.google.com/scholar?q=Ye+Qi%2C+Devendra+Singh+Sachan%2C+Matthieu+Felix%2C+Sarguna+Janani+Padmanabhan%2C+and+Graham+Neubig%2E+2018%2E+When+and+why+are+pre%2Dtrained+word+embeddings+useful+for+neural+machine+translation%3F+In+NAACL%2DHLT%2E) [our card](https://md.research.elamur.ai/free/en/papers/qi-2018-when-why-are-pre-trained/)
32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Jeffrey+Wu%2C+Rewon+Child%2C+David+Luan%2C+Dario+Amodei%2C+and+Ilya+Sutskever%2E+2019%2E+Language+models+are+unsupervised+multitask+learners%2E+Technical+report%2C+OpenAI%2E)
33. Anton Ragni, Edgar Dakin, Xie Chen, Mark JF Gales, and Kate M Knill. 2016. Multi-language neural network language models. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Anton+Ragni%2C+Edgar+Dakin%2C+Xie+Chen%2C+Mark+JF+Gales%2C+and+Kate+M+Knill%2E+2016%2E+Multi%2Dlanguage+neural+network+language+models%2E+In+INTERSPEECH%2E)
34. Prajit Ramachandran, Peter J Liu, and Quoc V Le. 2017. Unsupervised pretraining for sequence to sequence learning. In EMNLP. [find](https://scholar.google.com/scholar?q=Prajit+Ramachandran%2C+Peter+J+Liu%2C+and+Quoc+V+Le%2E+2017%2E+Unsupervised+pretraining+for+sequence+to+sequence+learning%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/ramachandran-2017-unsupervised-pretraining-sequence-sequence-learning/)
35. Brian Roark, Murat Saraclar, Michael Collins, and Mark Johnson. 2004. Discriminative language modeling with conditional random fields and the perceptron algorithm. In ACL. [find](https://scholar.google.com/scholar?q=Brian+Roark%2C+Murat+Saraclar%2C+Michael+Collins%2C+and+Mark+Johnson%2E+2004%2E+Discriminative+language+modeling+with+conditional+random+fields+and+the+perceptron+algorithm%2E+In+ACL%2E)
36. Carson T Schütze. 1996. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. Language Science Press. [find](https://scholar.google.com/scholar?q=Carson+T+Schu%CC%88tze%2E+1996%2E+The+empirical+base+of+linguistics%3A+Grammaticality+judgments+and+linguistic+methodology%2E+Language+Science+Press%2E)
37. Yangyang Shi, Martha Larson, Pascal Wiggers, and Catholijn M Jonker. 2013. Exploiting the succeeding words in recurrent neural network language models. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Yangyang+Shi%2C+Martha+Larson%2C+Pascal+Wiggers%2C+and+Catholijn+M+Jonker%2E+2013%2E+Exploiting+the+succeeding+words+in+recurrent+neural+network+language+models%2E+In+INTERSPEECH%2E)
38. Joongbo Shin, Yoonhyung Lee, and Kyomin Jung. 2019. Effective sentence scoring method using BERT for speech recognition. In ACML. [find](https://scholar.google.com/scholar?q=Joongbo+Shin%2C+Yoonhyung+Lee%2C+and+Kyomin+Jung%2E+2019%2E+Effective+sentence+scoring+method+using+BERT+for+speech+recognition%2E+In+ACML%2E)
39. Anuroop Sriram, Heewoo Jun, Sanjeev Satheesh, and Adam Coates. 2018. Cold fusion: Training seq2seq models together with language models. In INTER-SPEECH. [find](https://scholar.google.com/scholar?q=Anuroop+Sriram%2C+Heewoo+Jun%2C+Sanjeev+Satheesh%2C+and+Adam+Coates%2E+2018%2E+Cold+fusion%3A+Training+seq2seq+models+together+with+language+models%2E+In+INTER%2DSPEECH%2E)
40. Felix Stahlberg, James Cross, and Veselin Stoyanov. 2018. Simple fusion: Return of the language model. In WMT. [find](https://scholar.google.com/scholar?q=Felix+Stahlberg%2C+James+Cross%2C+and+Veselin+Stoyanov%2E+2018%2E+Simple+fusion%3A+Return+of+the+language+model%2E+In+WMT%2E)
41. Chenguang Wang, Mu Li, and Alexander J Smola. 2019. Language models with transformers. arXiv preprint arXiv:1904.09408. — [arXiv:1904.09408](https://arxiv.org/abs/1904.09408) [find](https://scholar.google.com/scholar?q=Chenguang+Wang%2C+Mu+Li%2C+and+Alexander+J+Smola%2E+2019%2E+Language+models+with+transformers%2E+arXiv+preprint+arXiv%3A1904%2E09408%2E+%E2%80%94+%5BarXiv%3A1904%2E09408%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1904%2E09408%29)
42. Xinyi Wang, Hieu Pham, Zihang Dai, and Graham Neubig. 2018. SwitchOut: An efficient data augmentation algorithm for neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Hieu+Pham%2C+Zihang+Dai%2C+and+Graham+Neubig%2E+2018%2E+SwitchOut%3A+An+efficient+data+augmentation+algorithm+for+neural+machine+translation%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2018-switchout-efficient-data-augmentation-algorithm/)
43. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. TACL. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Alicia+Parrish%2C+Haokun+Liu%2C+Anhad+Mohananey%2C+Wei+Peng%2C+Sheng%2DFu+Wang%2C+and+Samuel+R+Bowman%2E+2020%2E+BLiMP%3A+The+benchmark+of+linguistic+minimal+pairs+for+English%2E+TACL%2E)
44. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. TACL, 7:625–641. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2019%2E+Neural+network+acceptability+judgments%2E+TACL%2C+7%3A625%E2%80%93641%2E)
45. Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, et al. 2018. ESPnet: End-to-end speech processing toolkit. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Shinji+Watanabe%2C+Takaaki+Hori%2C+Shigeki+Karita%2C+Tomoki+Hayashi%2C+Jiro+Nishitoba%2C+Yuya+Unno%2C+Nelson+Enrique+Yalta+Soplin%2C+Jahn+Heymann%2C+Matthew+Wiesner%2C+Nanxin+Chen%2C+et+al%2E+2018%2E+ESPnet%3A+End%2Dto%2Dend+speec)
46. Thomas Wolf, L Debut, V Sanh, J Chaumond, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. 2019. HuggingFace’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. [find](https://scholar.google.com/scholar?q=Thomas+Wolf%2C+L+Debut%2C+V+Sanh%2C+J+Chaumond%2C+C+Delangue%2C+A+Moi%2C+P+Cistac%2C+T+Rault%2C+R+Louf%2C+M+Funtowicz%2C+et+al%2E+2019%2E+HuggingFace%E2%80%99s+transformers%3A+State%2Dof%2Dthe%2Dart+natural+language+processing%2E+arXiv+prepri) [our card](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/)
47. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
48. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In NeurIPS. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+Oriol+Vinyals%2C+and+Quoc+V+Le%2E+2014%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+NeurIPS%2E)
49. Deyi Xiong, Min Zhang, and Haizhou Li. 2011. Enhancing language models in statistical machine translation with backward n-grams and mutual information triggers. In ACL. [find](https://scholar.google.com/scholar?q=Deyi+Xiong%2C+Min+Zhang%2C+and+Haizhou+Li%2E+2011%2E+Enhancing+language+models+in+statistical+machine+translation+with+backward+n%2Dgrams+and+mutual+information+triggers%2E+In+ACL%2E)
50. Shuntaro Takahashi and Kumiko Tanaka-Ishii. 2018. Cross entropy of neural language models at infinity– a new bound of the entropy rate. Entropy, 20(11):839. [find](https://scholar.google.com/scholar?q=Shuntaro+Takahashi+and+Kumiko+Tanaka%2DIshii%2E+2018%2E+Cross+entropy+of+neural+language+models+at+infinity%E2%80%93+a+new+bound+of+the+entropy+rate%2E+Entropy%2C+20%2811%29%3A839%2E)
51. Yilin Yang, Liang Huang, and Mingbo Ma. 2018. Breaking the beam search curse: A study of (re-) scoring methods and stopping criteria for neural machine translation. In EMNLP. [find](https://scholar.google.com/scholar?q=Yilin+Yang%2C+Liang+Huang%2C+and+Mingbo+Ma%2E+2018%2E+Breaking+the+beam+search+curse%3A+A+study+of+%28re%2D%29+scoring+methods+and+stopping+criteria+for+neural+machine+translation%2E+In+EMNLP%2E)
52. Shubham Toshniwal, Anjuli Kannan, Chung-Cheng Chiu, Yonghui Wu, Tara N Sainath, and Karen Livescu. 2018. A comparison of techniques for language model integration in encoder-decoder speech recognition. In SLT. [find](https://scholar.google.com/scholar?q=Shubham+Toshniwal%2C+Anjuli+Kannan%2C+Chung%2DCheng+Chiu%2C+Yonghui+Wu%2C+Tara+N+Sainath%2C+and+Karen+Livescu%2E+2018%2E+A+comparison+of+techniques+for+language+model+integration+in+encoder%2Ddecoder+speech+recognition)
53. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS. [find](https://scholar.google.com/scholar?q=Zhilin+Yang%2C+Zihang+Dai%2C+Yiming+Yang%2C+Jaime+Carbonell%2C+Russ+R+Salakhutdinov%2C+and+Quoc+V+Le%2E+2019%2E+XLNet%3A+Generalized+autoregressive+pretraining+for+language+understanding%2E+In+NeurIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/yang-2019-xlnet-generalized-autoregressive-pretraining-language/)
54. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Zhaopeng+Tu%2C+Zhengdong+Lu%2C+Yang+Liu%2C+Xiaohua+Liu%2C+and+Hang+Li%2E+2016%2E+Modeling+coverage+for+neural+machine+translation%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/tu-2016-modeling-coverage-neural-machine-translation/)
55. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+%C5%81ukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NeurIPS%2E)
56. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In ICLR. [find](https://scholar.google.com/scholar?q=Tianyi+Zhang%2C+Varsha+Kishore%2C+Felix+Wu%2C+Kilian+Q+Weinberger%2C+and+Yoav+Artzi%2E+2020%2E+BERTScore%3A+Evaluating+text+generation+with+BERT%2E+In+ICLR%2E)
57. Jinhua Zhu, Yingce Xia, Lijun Wu, Di He, Tao Qin, Wengang Zhou, Houqiang Li, and Tie-Yan Liu. 2020. Incorporating BERT into neural machine translation. In ICLR. [find](https://scholar.google.com/scholar?q=Jinhua+Zhu%2C+Yingce+Xia%2C+Lijun+Wu%2C+Di+He%2C+Tao+Qin%2C+Wengang+Zhou%2C+Houqiang+Li%2C+and+Tie%2DYan+Liu%2E+2020%2E+Incorporating+BERT+into+neural+machine+translation%2E+In+ICLR%2E)
58. Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In NeuralGen. [find](https://scholar.google.com/scholar?q=Alex+Wang+and+Kyunghyun+Cho%2E+2019%2E+BERT+has+a+mouth%2C+and+it+must+speak%3A+BERT+as+a+Markov+random+field+language+model%2E+In+NeuralGen%2E) [our card](https://md.research.elamur.ai/free/en/papers/wang-2019-bert-has-mouth-must-speak/)