--- title: "How Multilingual is Multilingual BERT? — Pires 2019" description: "Most earlier probing work on contextual encoders looked only at what English models learn about English." authors: ["Telmo Pires", "Eva Schlinger", "Dan Garrette"] year: 2019 venue: "Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics" source: "https://aclanthology.org/P19-1493/" doi: "10.18653/v1/p19-1493" arxiv: "1906.01502" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / How Multilingual is Multilingual BERT? # How Multilingual is Multilingual BERT? **Authors:** Telmo Pires, Eva Schlinger, [Dan Garrette](https://md.research.elamur.ai/free/en/authors/dan-garrette/) · **Year:** 2019 · **Venue:** Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P19-1493/) · [PDF](https://aclanthology.org/P19-1493.pdf) · [DOI 10.18653/v1/p19-1493](https://doi.org/10.18653/v1/p19-1493) · [arXiv 1906.01502](https://arxiv.org/abs/1906.01502) · [OpenAlex](https://openalex.org/W2952638691) **Citations:** 1286 citations (openalex-snapshot · 2026-10-14) · [806](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Most earlier probing work on contextual encoders looked only at what English models learn about English. This paper tests how well Multilingual BERT (M-BERT) carries knowledge from one language to another. M-BERT is a single 12-layer transformer pre-trained on Wikipedia in 104 languages, with a shared word-piece vocabulary and no explicit cross-lingual signal. The authors fine-tune it for NER (CoNLL-2002/2003 plus an in-house 16-language set) or POS tagging (UD, 41 languages) in one language. They then evaluate it in another language, with no target-language labels. Further probes relate transfer quality to entity word-piece overlap, to WALS word-order features, to code-switched and transliterated Hindi/English, and to translation retrieval with sentence vectors on WMT16. Zero-shot transfer is strong and depends little on shared vocabulary. It works even between languages in different scripts, which points to real shared multilingual representations. However, it weakens when word order differs and breaks down on transliterated input. For lower-resource languages such as Ukrainian, this helps predict when labeled data from a related language is worth borrowing. ## Key points - **Little dependence on shared vocabulary:** Even when language pairs share almost no entity word pieces, M-BERT still gets roughly 40–70% NER F1. English-only BERT drops to near zero across scripts. - **Transfer across scripts:** After fine-tuning on Urdu POS data (Arabic script), M-BERT tags Hindi (Devanagari) with 91.1% accuracy. English→Bulgarian reaches 87.1%, but English→Japanese reaches only 49.4%. - **Word order matters:** Average POS accuracy is 81.55 for SVO→SVO transfer but 63.98 for SVO→SOV. More shared WALS features means better transfer. - **Code-switching vs. transliteration:** On the Hindi/English UD corpus of Bhat et al. (2018), training only on monolingual data gives 86.59% when Hindi is in Devanagari. Training on code-switched data gives 90.56%. With Hindi in Latin script, the monolingual-trained model falls to 50.41%. - **Translation retrieval without fine-tuning:** Shifting a sentence vector by one average offset per language pair finds the correct EN–DE translation more than 50% of the time in most layers. The first and last layers do worse. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 806 papers of our library cite it (of the 140673 whose bibliographies we read); 60 of them have a page here, 746 not yet. - [MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/adelani-2022-masakhaner-2-0-africa-centric/) - [Syntax-augmented Multilingual BERT for Cross-lingual Transfer](https://md.research.elamur.ai/free/en/papers/ahmad-2021-syntax-augmented-multilingual-bert-cross/) - [On the Cross-lingual Transferability of Monolingual Representations](https://md.research.elamur.ai/free/en/papers/artetxe-2020-cross-lingual-transferability-monolingual-representations/) - [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://md.research.elamur.ai/free/en/papers/barbieri-2022-xlm-t-multilingual-language-models/) - [ALLaM: Large Language Models for Arabic and English](https://md.research.elamur.ai/free/en/papers/bari-2024-allam-large-language-models-arabic/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Entity Linking in 100 Languages](https://md.research.elamur.ai/free/en/papers/botha-2020-entity-linking-100-languages/) - [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/chen-2024-m3-embedding-multi-linguality/) - [InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training](https://md.research.elamur.ai/free/en/papers/chi-2020-infoxlm-information-theoretic-framework-cross/) - [Rethinking embedding coupling in pre-trained language models](https://md.research.elamur.ai/free/en/papers/chung-2020-rethinking-embedding-coupling-pre-trained/) - [Emerging Cross-lingual Structure in Pretrained Language Models](https://md.research.elamur.ai/free/en/papers/conneau-2020-emerging-cross-lingual-structure-pretrained/) - [RobBERT: a Dutch RoBERTa-based Language Model](https://md.research.elamur.ai/free/en/papers/delobelle-2020-robbert-dutch-roberta-based-language/) - [AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages](https://md.research.elamur.ai/free/en/papers/ebrahimi-2022-americasnli-evaluating-zero-shot-natural/) - [BERT Fine-tuning For Arabic Text Summarization](https://md.research.elamur.ai/free/en/papers/elmadani-2020-bert-fine-tuning-arabic-text/) - [MuDRiC: Multi-Dialect Reasoning for Arabic Commonsense Validation](https://md.research.elamur.ai/free/en/papers/elozeiri-2025-mudric-multi-dialect-reasoning-arabic/) - [Filling the Gaps in Ancient Akkadian Texts: A Masked Language Modelling Approach](https://md.research.elamur.ai/free/en/papers/emnlp-2021-lazar-filling-gaps-ancient-akkadian-texts/) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://md.research.elamur.ai/free/en/papers/feng-2020-codebert-pre-trained-model-programming/) - [JABER and SABER: Junior and Senior Arabic BERt](https://md.research.elamur.ai/free/en/papers/ghaddar-2021-jaber-saber-junior-senior-arabic/) - [AmaSQuAD: A Benchmark for Amharic Extractive Question Answering](https://md.research.elamur.ai/free/en/papers/hailemariam-2025-amasquad-benchmark-amharic-extractive-question/) - [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://md.research.elamur.ai/free/en/papers/hessel-2021-clipscore-reference-free-evaluation-metric/) - [Bridging the Resource Gap: Exploring the Efficacy of English and Multilingual LLMs for Swedish](https://md.research.elamur.ai/free/en/papers/holmstrom-2023-bridging-resource-gap-exploring-efficacy/) - [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://md.research.elamur.ai/free/en/papers/hu-2020-xtreme-massively-multilingual-multi-task/) - [The State and Fate of Linguistic Diversity and Inclusion in the NLP World](https://md.research.elamur.ai/free/en/papers/joshi-2020-state-fate-linguistic-diversity-inclusion/) - [IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages](https://md.research.elamur.ai/free/en/papers/kakwani-2020-indicnlpsuite-monolingual-corpora-evaluation-benchmarks/) - [GLUECoS: An Evaluation Benchmark for Code-Switched NLP](https://md.research.elamur.ai/free/en/papers/khanuja-2020-gluecos-evaluation-benchmark-code-switched/) - [From Zero to Hero: On the Limitations of Zero-Shot Language Transfer with Multilingual Transformers](https://md.research.elamur.ai/free/en/papers/lauscher-2020-zero-hero-limitations-zero-shot/) - [MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmark](https://md.research.elamur.ai/free/en/papers/li-2021-mtop-comprehensive-multilingual-task-oriented/) - [Few-shot Learning with Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/lin-2021-few-shot-learning-multilingual-language/) - [Playing with Words at the National Library of Sweden -- Making a Swedish BERT](https://md.research.elamur.ai/free/en/papers/malmsten-2020-playing-words-national-library-sweden/) - [CamemBERT: a Tasty French Language Model](https://md.research.elamur.ai/free/en/papers/martin-2019-camembert-tasty-french-language-model/) - [Exploring Pre-Trained Transformers and Bilingual Transfer Learning for Arabic Coreference Resolution](https://md.research.elamur.ai/free/en/papers/min-2021-exploring-pre-trained-transformers-bilingual/) - [Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi](https://md.research.elamur.ai/free/en/papers/muller-2020-multilingual-language-models-transfer-unseen/) - [When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/muller-2021-when-being-unseen-mbert-is/) - [iCompass at Shared Task on Sarcasm and Sentiment Detection in Arabic](https://md.research.elamur.ai/free/en/papers/naski-2021-icompass-shared-task-sarcasm-sentiment/) - [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://md.research.elamur.ai/free/en/papers/nekoto-2020-participatory-research-low-resourced-machine/) - [SemEval-2020 Task 9: Overview of Sentiment Analysis of Code-Mixed Tweets](https://md.research.elamur.ai/free/en/papers/patwa-2020-semeval-2020-task-9-overview/) - [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://md.research.elamur.ai/free/en/papers/pfeiffer-2020-mad-x-adapter-based-framework/) - [UNKs Everywhere: Adapting Multilingual Language Models to New Scripts](https://md.research.elamur.ai/free/en/papers/pfeiffer-2021-unks-everywhere-adapting-multilingual-language/) - [Lifting the Curse of Multilinguality by Pre-training Modular Transformers](https://md.research.elamur.ai/free/en/papers/pfeiffer-2022-lifting-curse-multilinguality-pre-training/) - [Learning Compact Metrics for MT](https://md.research.elamur.ai/free/en/papers/pu-2021-learning-compact-metrics-mt/) - [Pre-trained Models for Natural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/qiu-2020-pre-trained-models-natural-language/) - [Under the Surface: Probing Tamil Paraphrase Intelligence](https://md.research.elamur.ai/free/en/papers/r-2026-under-surface-probing-tamil-paraphrase/) - [TransQuest: Translation Quality Estimation with Cross-lingual Transformers](https://md.research.elamur.ai/free/en/papers/ranasinghe-2020-transquest-translation-quality-estimation-cross/) - [COMET: A Neural Framework for MT Evaluation](https://md.research.elamur.ai/free/en/papers/rei-2020-comet-neural-framework-mt-evaluation/) - [How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models](https://md.research.elamur.ai/free/en/papers/rust-2021-how-good-is-your-tokenizer/) - [Parameter and Data Efficient Continual Pre-training for Robustness to Dialectal Variance in Arabic](https://md.research.elamur.ai/free/en/papers/sarkar-2022-parameter-data-efficient-continual-pre/) - [MLSUM: The Multilingual Summarization Corpus](https://md.research.elamur.ai/free/en/papers/scialom-2020-mlsum-multilingual-summarization-corpus/) - [Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task](https://md.research.elamur.ai/free/en/papers/sellam-2020-learning-evaluate-translation-beyond-english/) - [Language Models are Multilingual Chain-of-Thought Reasoners](https://md.research.elamur.ai/free/en/papers/shi-2022-language-models-multilingual-chain-thought/) - [Is Cross-Lingual Transfer in Bilingual Models Human-Like? A Study with Overlapping Word Forms in Dutch and English](https://md.research.elamur.ai/free/en/papers/skrjanec-2026-is-cross-lingual-transfer-bilingual/) - [Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models](https://md.research.elamur.ai/free/en/papers/tang-2024-language-specific-neurons-key-multilingual/) - [How Are Idioms Processed Inside Transformer Language Models?](https://md.research.elamur.ai/free/en/papers/tian-2023-how-are-idioms-processed-inside/) - [Multilingual is not enough: BERT for Finnish](https://md.research.elamur.ai/free/en/papers/virtanen-2019-multilingual-enough-bert-finnish/) - [On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment](https://md.research.elamur.ai/free/en/papers/wang-2020-negative-interference-multilingual-models-findings/) - [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://md.research.elamur.ai/free/en/papers/wu-2019-beto-bentz-becas-surprising-cross/) - [Are All Languages Created Equal in Multilingual BERT?](https://md.research.elamur.ai/free/en/papers/wu-2020-are-all-languages-created-equal/) - [End-to-End Slot Alignment and Recognition for Cross-Lingual NLU](https://md.research.elamur.ai/free/en/papers/xu-2020-end-end-slot-alignment-recognition/) - [MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages](https://md.research.elamur.ai/free/en/papers/zhang-2023-miracl-multilingual-retrieval-dataset-covering/) - [YNU-HPCC at SemEval-2022 Task 6: Transformer-based Model for Intended Sarcasm Detection in English and Arabic](https://md.research.elamur.ai/free/en/papers/zheng-2022-ynu-hpcc-semeval-2022-task/) - [Multi-VALUE: A Framework for Cross-Dialectal English NLP](https://md.research.elamur.ai/free/en/papers/ziems-2023-multi-value-framework-cross-dialectal/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{pires2019how, title = {{How Multilingual is Multilingual BERT?}}, author = {Pires, Telmo and Schlinger, Eva and Garrette, Dan}, booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics}, year = {2019}, doi = {10.18653/v1/p19-1493}, eprint = {1906.01502}, archivePrefix = {arXiv}, url = {https://doi.org/10.18653/v1/p19-1493} } ``` - [← previous Historical Text Normalization with Delayed Rewards](https://md.research.elamur.ai/free/en/papers/acl-2019-flachs-historical-text-normalization-delayed-rewards/) - [next → How to Fine-Tune BERT for Text Classification?](https://md.research.elamur.ai/free/en/papers/sun-2019-fine-tune-bert-text-classification/) - [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) ## References 15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Mikel Artetxe and Holger Schwenk. 2018. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. arXiv preprint arXiv:1812.10464. [find](https://scholar.google.com/scholar?q=Mikel+Artetxe+and+Holger+Schwenk%2E+2018%2E+Massively+multilingual+sentence+embeddings+for+zero%2Dshot+cross%2Dlingual+transfer+and+beyond%2E+arXiv+preprint+arXiv%3A1812%2E10464%2E) [our card](https://md.research.elamur.ai/free/en/papers/artetxe-2018-massively-multilingual-sentence-embeddings-zero/) 2. Kelsey Ball and Dan Garrette. 2018. Part-of-speech tagging for code-switched, transliterated texts without explicit language identification. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Kelsey+Ball+and+Dan+Garrette%2E+2018%2E+Part%2Dof%2Dspeech+tagging+for+code%2Dswitched%2C+transliterated+texts+without+explicit+language+identification%2E+In+Proceedings+of+EMNLP%2E) 3. Irshad Bhat, Riyaz A. Bhat, Manish Shrivastava, and Dipti Sharma. 2018. Universal dependency parsing for Hindi-English code-switching. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Irshad+Bhat%2C+Riyaz+A%2E+Bhat%2C+Manish+Shrivastava%2C+and+Dipti+Sharma%2E+2018%2E+Universal+dependency+parsing+for+Hindi%2DEnglish+code%2Dswitching%2E+In+Proceedings+of+NAACL%2E) 4. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 metrics shared task. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers. [find](https://scholar.google.com/scholar?q=Ondr%CC%8Cej+Bojar%2C+Yvette+Graham%2C+Amir+Kamran%2C+and+Milos%CC%8C+Stanojevic%CC%81%2E+2016%2E+Results+of+the+WMT16+metrics+shared+task%2E+In+Proceedings+of+the+First+Conference+on+Machine+Translation%3A+Volume+2%2C+Shared+Task+) [our card](https://md.research.elamur.ai/free/en/papers/bojar-2016-results-wmt16-metrics-shared-task/) 5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2019%2E+BERT%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E+In+Proceedings+of+NAACL%2E) 6. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. [https://wals.info/](https://wals.info/). [↗](https://wals.info/) 7. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Miguel+Ballesteros%2C+Sandeep+Subramanian%2C+Kazuya+Kawakami%2C+and+Chris+Dyer%2E+2016%2E) 8. Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonca, Tatiana Lando, Rattima Nitisaroj, and Josie Li… [find](https://scholar.google.com/scholar?q=Kayadelen%2C+Mohammed+Attia%2C+Ali+Elkahky%2C+Zhuoran+Yu%2C+Emily+Pitler%2C+Saran+Lertpradit%2C+Michael+Mandl%2C+Jesse+Kirchner%2C+Hector+Fernandez+Alcalde%2C+Jana+Strnadova%CC%81%2C+Esha+Banerjee%2C+Ruli+Manurung%2C+Antonio+Stel) 9. CoNLL 2017 shared task: Multilingual parsing from raw text to universal dependencies. In Proceedings of CoNLL. [find](https://scholar.google.com/scholar?q=CoNLL+2017+shared+task%3A+Multilingual+parsing+from+raw+text+to+universal+dependencies%2E+In+Proceedings+of+CoNLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/ud-2017-zeman-conll-2017-shared-task-multilingual/) 10. Neural architectures for named entity recognition. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Neural+architectures+for+named+entity+recognition%2E+In+Proceedings+of+NAACL%2E) 11. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. [find](https://scholar.google.com/scholar?q=Guillaume+Lample+and+Alexis+Conneau%2E+2019%2E+Cross%2Dlingual+language+model+pretraining%2E+arXiv+preprint+arXiv%3A1901%2E07291%2E) 12. Tahira Naseem, Regina Barzilay, and Amir Globerson. 2012. Selective sharing for multilingual dependency parsing. In Proceedings of ACL. [find](https://scholar.google.com/scholar?q=Tahira+Naseem%2C+Regina+Barzilay%2C+and+Amir+Globerson%2E+2012%2E+Selective+sharing+for+multilingual+dependency+parsing%2E+In+Proceedings+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/naseem-2012-selective-sharing-multilingual-dependency-parsing/) 13. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D. Manning, Ryan T. McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of LREC. [find](https://scholar.google.com/scholar?q=Joakim+Nivre%2C+Marie%2DCatherine+de+Marneffe%2C+Filip+Ginter%2C+Yoav+Goldberg%2C+Jan+Hajic%2C+Christopher+D%2E+Manning%2C+Ryan+T%2E+McDonald%2C+Slav+Petrov%2C+Sampo+Pyysalo%2C+Natalia+Silveira%2C+Reut+Tsarfaty%2C+and+Daniel+Zem) [our card](https://md.research.elamur.ai/free/en/papers/ud-2016-nivre-universal-dependencies-v1-multilingual-treebank/) 14. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018a. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of EMNLP. [find](https://scholar.google.com/scholar?q=Matthew+Peters%2C+Mark+Neumann%2C+Luke+Zettlemoyer%2C+and+Wen%2Dtau+Yih%2E+2018a%2E+Dissecting+contextual+word+embeddings%3A+Architecture+and+representation%2E+In+Proceedings+of+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/) 15. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018b. Deep contextualized word representations. In Proceedings of NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018b%2E+Deep+contextualized+word+representations%2E+In+Proceedings+of+NAACL%2E)