---
title: "BERT: Pre-training of Deep Bidirectional — Devlin 2019"
description: "The authors propose BERT, a language representation model based on the Transformer encoder."
authors: ["Jacob Devlin", "Ming-Wei Chang", "Kenton Lee", "Kristina Toutanova"]
year: 2019
venue: "Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)"
source: "https://aclanthology.org/N19-1423/"
doi: "10.18653/v1/n19-1423"
arxiv: "1810.04805"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/) / BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

# BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

**Authors:** [Jacob Devlin](https://md.research.elamur.ai/free/en/authors/jacob-devlin/), [Ming-Wei Chang](https://md.research.elamur.ai/free/en/authors/ming-wei-chang/), [Kenton Lee](https://md.research.elamur.ai/free/en/authors/kenton-lee/), [Kristina Toutanova](https://md.research.elamur.ai/free/en/authors/kristina-toutanova/) · **Year:** 2019 · **Venue:** Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N19-1423/) · [PDF](https://aclanthology.org/N19-1423.pdf) · [DOI 10.18653/v1/n19-1423](https://doi.org/10.18653/v1/n19-1423) · [arXiv 1810.04805](https://arxiv.org/abs/1810.04805)

**Citations:** [23422](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
The authors propose BERT, a language representation model based on the Transformer encoder. It is pre-trained on unlabeled text in such a way that every layer sees the context both to the left and to the right of a word. Previous approaches had limitations. OpenAI GPT reads text in only one direction. ELMo simply combines two separately trained unidirectional models. As a result, quality suffered, especially on token-level tasks. BERT solves this problem with two pre-training tasks. The first is the masked language model (MLM): some of the tokens are hidden, and the model reconstructs them from their surroundings. The second is next sentence prediction (NSP), which teaches relationships between pairs of sentences. The same model is then fine-tuned for a specific task by adding only a simple output layer. BERT achieved new state-of-the-art results on eleven NLP tasks, including GLUE, MultiNLI and SQuAD. The work is important because a single general-purpose pre-trained model replaced many specially designed architectures. The code and weights were released publicly, and the approach became the standard in the field.

## Key points
- There are two configurations. BERT BASE has L=12, H=768, A=12 and 110M parameters; its size was deliberately chosen to match OpenAI GPT. BERT LARGE has L=24, H=1024, A=16 and 340M parameters.
- The input uses a WordPiece vocabulary of 30,000 tokens and the special tokens [CLS] (for classification) and [SEP] (separator). The representation of each token is the sum of the token, segment and position embeddings.
- Results and improvement over the previous state of the art: GLUE — 80.5 (+7.7 points), MultiNLI — 86.7% (+4.6), SQuAD v1.1 Test F1 — 93.2 (+1.5), SQuAD v2.0 Test F1 — 83.1 (+5.1).
- A larger model yields higher accuracy even on small datasets such as MRPC (about 3,600 training examples), provided the model has been sufficiently pre-trained.
- On CoNLL-2003 NER, a variant without fine-tuning was also tested, in which BERT only produces features for another model. Concatenating the top four layers of BERT BASE gives a Dev F1 of 96.1, only 0.3 lower than full fine-tuning (96.4). Code and models: github.com/google-research/bert.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{devlin2019bert,
  title = {{BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding}},
  author = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1423},
  eprint = {1810.04805},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1423}
}
```

- [← previous SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python](https://md.research.elamur.ai/free/en/papers/virtanen-2019-scipy-1-0-fundamental-algorithms/)
- [next → PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://md.research.elamur.ai/free/en/papers/paszke-2019-pytorch-imperative-style-high-performance/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/)

## References

54 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638–1649. [find](https://scholar.google.com/scholar?q=Alan+Akbik%2C+Duncan+Blythe%2C+and+Roland+Vollgraf%2E+2018%2E+Contextual+string+embeddings+for+sequence+labeling%2E+In+Proceedings+of+the+27th+International+Conference+on+Computational+Linguistics%2C+pages+1638%E2%80%931) [our card](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/)
2. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2018. Character-level language modeling with deeper self-attention. arXiv preprint arXiv:1808.04444. [find](https://scholar.google.com/scholar?q=Rami+Al%2DRfou%2C+Dokook+Choe%2C+Noah+Constant%2C+Mandy+Guo%2C+and+Llion+Jones%2E+2018%2E+Character%2Dlevel+language+modeling+with+deeper+self%2Dattention%2E+arXiv+preprint+arXiv%3A1808%2E04444%2E)
3. Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(Nov):1817–1853. [find](https://scholar.google.com/scholar?q=Rie+Kubota+Ando+and+Tong+Zhang%2E+2005%2E+A+framework+for+learning+predictive+structures+from+multiple+tasks+and+unlabeled+data%2E+Journal+of+Machine+Learning+Research%2C+6%28Nov%29%3A1817%E2%80%931853%2E)
4. Luisa Bentivogli, Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC. NIST. [find](https://scholar.google.com/scholar?q=Luisa+Bentivogli%2C+Bernardo+Magnini%2C+Ido+Dagan%2C+Hoa+Trang+Dang%2C+and+Danilo+Giampiccolo%2E+2009%2E+The+fifth+PASCAL+recognizing+textual+entailment+challenge%2E+In+TAC%2E+NIST%2E)
5. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 120–128. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=John+Blitzer%2C+Ryan+McDonald%2C+and+Fernando+Pereira%2E+2006%2E+Domain+adaptation+with+structural+correspondence+learning%2E+In+Proceedings+of+the+2006+conference+on+empirical+methods+in+natural+language+proce) [our card](https://md.research.elamur.ai/free/en/papers/blitzer-2006-domain-adaptation-structural-correspondence-learning/)
6. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+EMNLP%2E+Association+for+Computational+Linguisti)
7. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, and Jenifer C Lai. 1992. Class-based n-gram models of natural language. Computational linguistics, 18(4):467–479. [find](https://scholar.google.com/scholar?q=Peter+F+Brown%2C+Peter+V+Desouza%2C+Robert+L+Mercer%2C+Vincent+J+Della+Pietra%2C+and+Jenifer+C+Lai%2E+1992%2E+Class%2Dbased+n%2Dgram+models+of+natural+language%2E+Computational+linguistics%2C+18%284%29%3A467%E2%80%93479%2E)
8. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada.… [find](https://scholar.google.com/scholar?q=Daniel+Cer%2C+Mona+Diab%2C+Eneko+Agirre%2C+Inigo+Lopez%2DGazpio%2C+and+Lucia+Specia%2E+2017%2E+Semeval%2D2017+task+1%3A+Semantic+textual+similarity+multilingual+and+crosslingual+focused+evaluation%2E+In+Proceedings+of+th) [our card](https://md.research.elamur.ai/free/en/papers/cer-2017-semeval-2017-task-1-semantic/)
9. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005. [find](https://scholar.google.com/scholar?q=Ciprian+Chelba%2C+Tomas+Mikolov%2C+Mike+Schuster%2C+Qi+Ge%2C+Thorsten+Brants%2C+Phillipp+Koehn%2C+and+Tony+Robinson%2E+2013%2E+One+billion+word+benchmark+for+measuring+progress+in+statistical+language+modeling%2E+arXiv) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
10. Z. Chen, H. Zhang, X. Zhang, and L. Zhao. 2018. Quora question pairs. [find](https://scholar.google.com/scholar?q=Z%2E+Chen%2C+H%2E+Zhang%2C+X%2E+Zhang%2C+and+L%2E+Zhao%2E+2018%2E+Quora+question+pairs%2E)
11. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Christopher+Clark+and+Matt+Gardner%2E+2018%2E+Simple+and+effective+multi%2Dparagraph+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)
12. Kevin Clark, Minh-Thang Luong, Christopher D Manning, and Quoc Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1914– 1925. [find](https://scholar.google.com/scholar?q=Kevin+Clark%2C+Minh%2DThang+Luong%2C+Christopher+D+Manning%2C+and+Quoc+Le%2E+2018%2E+Semi%2Dsupervised+sequence+modeling+with+cross%2Dview+training%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natur) [our card](https://md.research.elamur.ai/free/en/papers/clark-2018-semi-supervised-sequence-modeling-cross/)
13. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. [find](https://scholar.google.com/scholar?q=Ronan+Collobert+and+Jason+Weston%2E+2008%2E+A+unified+architecture+for+natural+language+processing%3A+Deep+neural+networks+with+multitask+learning%2E+In+Proceedings+of+the+25th+international+conference+on+Mac)
14. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,… [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+Douwe+Kiela%2C+Holger+Schwenk%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Antoine+Bordes%2E+2017%2E+Supervised+learning+of+universal+sentence+representations+from+natural+language+inference+data%2E+In+Proceedings+of) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2017-supervised-learning-universal-sentence-representations/)
15. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087. [find](https://scholar.google.com/scholar?q=Andrew+M+Dai+and+Quoc+V+Le%2E+2015%2E+Semi%2Dsupervised+sequence+learning%2E+In+Advances+in+neural+information+processing+systems%2C+pages+3079%E2%80%933087%2E)
16. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09. [find](https://scholar.google.com/scholar?q=J%2E+Deng%2C+W%2E+Dong%2C+R%2E+Socher%2C+L%2E%2DJ%2E+Li%2C+K%2E+Li%2C+and+L%2E+Fei%2DFei%2E+2009%2E+ImageNet%3A+A+Large%2DScale+Hierarchical+Image+Database%2E+In+CVPR09%2E)
17. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). [find](https://scholar.google.com/scholar?q=William+B+Dolan+and+Chris+Brockett%2E+2005%2E+Automatically+constructing+a+corpus+of+sentential+paraphrases%2E+In+Proceedings+of+the+Third+International+Workshop+on+Paraphrasing+%28IWP2005%29%2E) [our card](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/)
18. William Fedus, Ian Goodfellow, and Andrew M Dai. 2018. Maskgan: Better text generation via filling in the . arXiv preprint arXiv:1801.07736. — [arXiv:1801.07736](https://arxiv.org/abs/1801.07736) [find](https://scholar.google.com/scholar?q=William+Fedus%2C+Ian+Goodfellow%2C+and+Andrew+M+Dai%2E+2018%2E+Maskgan%3A+Better+text+generation+via+filling+in+the+%2E+arXiv+preprint+arXiv%3A1801%2E07736%2E+%E2%80%94+%5BarXiv%3A1801%2E07736%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1801%2E07736%29)
19. Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415. [find](https://scholar.google.com/scholar?q=Dan+Hendrycks+and+Kevin+Gimpel%2E+2016%2E+Bridging+nonlinearities+and+stochastic+regularizers+with+gaussian+error+linear+units%2E+CoRR%2C+abs%2F1606%2E08415%2E)
20. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational… [find](https://scholar.google.com/scholar?q=Felix+Hill%2C+Kyunghyun+Cho%2C+and+Anna+Korhonen%2E+2016%2E+Learning+distributed+representations+of+sentences+from+unlabelled+data%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+A) [our card](https://md.research.elamur.ai/free/en/papers/hill-2016-learning-distributed-representations-sentences-unlabelled/)
21. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E+Association+for+Computational+Linguistics%2E)
22. Minghao Hu, Yuxing Peng, Zhen Huang, Xipeng Qiu, Furu Wei, and Ming Zhou. 2018. Reinforced mnemonic reader for machine reading comprehension. In IJCAI. [find](https://scholar.google.com/scholar?q=Minghao+Hu%2C+Yuxing+Peng%2C+Zhen+Huang%2C+Xipeng+Qiu%2C+Furu+Wei%2C+and+Ming+Zhou%2E+2018%2E+Reinforced+mnemonic+reader+for+machine+reading+comprehension%2E+In+IJCAI%2E)
23. Yacine Jernite, Samuel R. Bowman, and David Sontag. 2017. Discourse-based objectives for fast unsupervised sentence representation learning. CoRR, abs/1705.00557. — [arXiv:1705.00557](https://arxiv.org/abs/1705.00557) [find](https://scholar.google.com/scholar?q=Yacine+Jernite%2C+Samuel+R%2E+Bowman%2C+and+David+Sontag%2E+2017%2E+Discourse%2Dbased+objectives+for+fast+unsupervised+sentence+representation+learning%2E+CoRR%2C+abs%2F1705%2E00557%2E+%E2%80%94+%5BarXiv%3A1705%2E00557%5D%28https%3A%2F%2Farxiv%2Eor)
24. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. [find](https://scholar.google.com/scholar?q=Mandar+Joshi%2C+Eunsol+Choi%2C+Daniel+S+Weld%2C+and+Luke+Zettlemoyer%2E+2017%2E+Triviaqa%3A+A+large+scale+distantly+supervised+challenge+dataset+for+reading+comprehension%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/joshi-2017-triviaqa-large-scale-distantly-supervised/)
25. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. [find](https://scholar.google.com/scholar?q=Ryan+Kiros%2C+Yukun+Zhu%2C+Ruslan+R+Salakhutdinov%2C+Richard+Zemel%2C+Raquel+Urtasun%2C+Antonio+Torralba%2C+and+Sanja+Fidler%2E+2015%2E+Skip%2Dthought+vectors%2E+In+Advances+in+neural+information+processing+systems%2C+page)
26. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning, pages 1188–1196. [find](https://scholar.google.com/scholar?q=Quoc+Le+and+Tomas+Mikolov%2E+2014%2E+Distributed+representations+of+sentences+and+documents%2E+In+International+Conference+on+Machine+Learning%2C+pages+1188%E2%80%931196%2E) [our card](https://md.research.elamur.ai/free/en/papers/le-2014-distributed-representations-sentences-documents/)
27. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The winograd schema challenge. In Aaai spring symposium: Logical formalizations of commonsense reasoning, volume 46, page 47. [find](https://scholar.google.com/scholar?q=Hector+J+Levesque%2C+Ernest+Davis%2C+and+Leora+Morgenstern%2E+2011%2E+The+winograd+schema+challenge%2E+In+Aaai+spring+symposium%3A+Logical+formalizations+of+commonsense+reasoning%2C+volume+46%2C+page+47%2E)
28. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Lajanugen+Logeswaran+and+Honglak+Lee%2E+2018%2E+An+efficient+framework+for+learning+sentence+representations%2E+In+International+Conference+on+Learning+Representations%2E) [our card](https://md.research.elamur.ai/free/en/papers/logeswaran-2018-efficient-framework-learning-sentence-representations/)
29. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+James+Bradbury%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+NIPS%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
30. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In CoNLL. [find](https://scholar.google.com/scholar?q=Oren+Melamud%2C+Jacob+Goldberger%2C+and+Ido+Dagan%2E+2016%2E+context2vec%3A+Learning+generic+context+embedding+with+bidirectional+LSTM%2E+In+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/melamud-2016-context2vec-learning-generic-context-embedding/)
31. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Ilya+Sutskever%2C+Kai+Chen%2C+Greg+S+Corrado%2C+and+Jeff+Dean%2E+2013%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+Neural+Information+Processing+) [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
32. Andriy Mnih and Geoffrey E Hinton. 2009. A scalable hierarchical distributed language model. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 1081–1088. Curran Associates, Inc. [find](https://scholar.google.com/scholar?q=Andriy+Mnih+and+Geoffrey+E+Hinton%2E+2009%2E+A+scalable+hierarchical+distributed+language+model%2E+In+D%2E+Koller%2C+D%2E+Schuurmans%2C+Y%2E+Bengio%2C+and+L%2E+Bottou%2C+editors%2C+Advances+in+Neural+Information+Processing+S)
33. Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP. [find](https://scholar.google.com/scholar?q=Ankur+P+Parikh%2C+Oscar+Ta%CC%88ckstro%CC%88m%2C+Dipanjan+Das%2C+and+Jakob+Uszkoreit%2E+2016%2E+A+decomposable+attention+model+for+natural+language+inference%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/parikh-2016-decomposable-attention-model-natural-language/)
34. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1532%E2%80%93+1543%2E) [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
35. Matthew Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Waleed+Ammar%2C+Chandra+Bhagavatula%2C+and+Russell+Power%2E+2017%2E+Semi%2Dsupervised+sequence+tagging+with+bidirectional+language+models%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/)
36. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In NAACL. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018a%2E+Deep+contextualized+word+representations%2E+In+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)
37. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Luke+Zettlemoyer%2C+and+Wen%2Dtau+Yih%2E+2018b%2E+Dissecting+contextual+word+embeddings%3A+Architecture+and+representation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Metho) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)
38. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, OpenAI. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+with+unsupervised+learning%2E+Technical+report%2C+OpenAI%2E)
39. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. [find](https://scholar.google.com/scholar?q=Pranav+Rajpurkar%2C+Jian+Zhang%2C+Konstantin+Lopyrev%2C+and+Percy+Liang%2E+2016%2E+Squad%3A+100%2C000%2B+questions+for+machine+comprehension+of+text%2E+In+Proceedings+of+the+2016+Conference+on+Empirical+Methods+in+Natu) [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
40. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. [find](https://scholar.google.com/scholar?q=Minjoon+Seo%2C+Aniruddha+Kembhavi%2C+Ali+Farhadi%2C+and+Hannaneh+Hajishirzi%2E+2017%2E+Bidirectional+attention+flow+for+machine+comprehension%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
41. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E+In+)
42. Fu Sun, Linyang Li, Xipeng Qiu, and Yang Liu. 2018. U-net: Machine reading comprehension with unanswerable questions. arXiv preprint arXiv:1810.06638. — [arXiv:1810.06638](https://arxiv.org/abs/1810.06638) [find](https://scholar.google.com/scholar?q=Fu+Sun%2C+Linyang+Li%2C+Xipeng+Qiu%2C+and+Yang+Liu%2E+2018%2E+U%2Dnet%3A+Machine+reading+comprehension+with+unanswerable+questions%2E+arXiv+preprint+arXiv%3A1810%2E06638%2E+%E2%80%94+%5BarXiv%3A1810%2E06638%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1810%2E0)
43. Wilson L Taylor. 1953. Cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. [find](https://scholar.google.com/scholar?q=Wilson+L+Taylor%2E+1953%2E+Cloze+procedure%3A+A+new+tool+for+measuring+readability%2E+Journalism+Bulletin%2C+30%284%29%3A415%E2%80%93433%2E)
44. Erik F Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In CoNLL. [find](https://scholar.google.com/scholar?q=Erik+F+Tjong+Kim+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+conll%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/)
45. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL ’10, pages 384–394. [find](https://scholar.google.com/scholar?q=Joseph+Turian%2C+Lev+Ratinov%2C+and+Yoshua+Bengio%2E+2010%2E+Word+representations%3A+A+simple+and+general+method+for+semi%2Dsupervised+learning%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Co) [our card](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)
46. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 6000–6010. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+Advances+in+Neural+Information+Processi) [our card](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/)
47. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096–1103. ACM. [find](https://scholar.google.com/scholar?q=Pascal+Vincent%2C+Hugo+Larochelle%2C+Yoshua+Bengio%2C+and+Pierre%2DAntoine+Manzagol%2E+2008%2E+Extracting+and+composing+robust+features+with+denoising+autoencoders%2E+In+Proceedings+of+the+25th+international+confer)
48. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018a. Glue: A multi-task benchmark and analysis platform • Additional details for our experiments are presented in Appendix B; and for natural language understanding. In Proceedings of the 2018 EMNLP Workshop Blac… [find](https://scholar.google.com/scholar?q=Alex+Wang%2C+Amanpreet+Singh%2C+Julian+Michael%2C+Felix+Hill%2C+Omer+Levy%2C+and+Samuel+Bowman%2E+2018a%2E+Glue%3A+A+multi%2Dtask+benchmark+and+analysis+platform+%E2%80%A2+Additional+details+for+our+experiments+are+presented+i)
49. Additional ablation studies are presented in Appendix C. [find](https://scholar.google.com/scholar?q=Additional+ablation+studies+are+presented+in+Appendix+C%2E)
50. Wei Wang, Ming Yan, and Chen Wu. 2018b. Multi-granularity hierarchical attention fusion networks for reading comprehension and question answering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Wei+Wang%2C+Ming+Yan%2C+and+Chen+Wu%2E+2018b%2E+Multi%2Dgranularity+hierarchical+attention+fusion+networks+for+reading+comprehension+and+question+answering%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Asso)
51. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. [find](https://scholar.google.com/scholar?q=Alex+Warstadt%2C+Amanpreet+Singh%2C+and+Samuel+R+Bowman%2E+2018%2E+Neural+network+acceptability+judgments%2E+arXiv+preprint+arXiv%3A1805%2E12471%2E)
52. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+R+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2E)
53. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+et+al%2E+2016%2E+Google%E2%80%99s+neural+machine+translation+system%3A+Bridg)
54. We present additional ablation studies for BERT including: – Effect of Number of Training Steps; and – Ablation for Different Masking Procedures. [find](https://scholar.google.com/scholar?q=We+present+additional+ablation+studies+for+BERT+including%3A+%E2%80%93+Effect+of+Number+of+Training+Steps%3B+and+%E2%80%93+Ablation+for+Different+Masking+Procedures%2E)