---
title: "Dissecting Contextual Word Embeddings — Peters 2018"
description: "Pretrained bidirectional language models (biLMs) such as ELMo improve many NLP tasks, but it was unclear whether the usual LSTM backbone matters or what…"
authors: ["Matthew E. Peters", "Mark Neumann", "Luke Zettlemoyer", "Wen-tau Yih"]
year: 2018
venue: "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D18-1179/"
doi: "10.18653/v1/d18-1179"
arxiv: "1808.08949"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peters-2018-dissecting-contextual-word-embeddings-architecture/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Dissecting Contextual Word Embeddings: Architecture and Representation

# Dissecting Contextual Word Embeddings: Architecture and Representation

**Authors:** [Matthew E. Peters](https://md.research.elamur.ai/free/en/authors/matthew-e-peters/), [Mark Neumann](https://md.research.elamur.ai/free/en/authors/mark-neumann/), [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/), [Wen-tau Yih](https://md.research.elamur.ai/free/en/authors/wen-tau-yih/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D18-1179/) · [PDF](https://aclanthology.org/D18-1179.pdf) · [DOI 10.18653/v1/d18-1179](https://doi.org/10.18653/v1/d18-1179) · [arXiv 1808.08949](https://arxiv.org/abs/1808.08949) · [OpenAlex](https://openalex.org/W2888329843)

**Citations:** 426 citations (openalex-snapshot · 2026-10-14) · [159](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Pretrained bidirectional language models (biLMs) such as ELMo improve many NLP tasks, but it was unclear whether the usual LSTM backbone matters or what the layers actually encode. The authors trained biLMs on the 1 Billion Word Benchmark with three architectures for the contextual layers: LSTM (2 and 4 layers), a 6-layer Transformer and a 16-layer Gated CNN. All models share a character-based word encoder and reach roughly comparable perplexity. They fed ELMo-style weighted layer mixtures into strong models for MultiNLI, SRL, constituency parsing and NER. They also probed individual layers with linear classifiers for syntactic and semantic properties.

Every architecture clearly beat GloVe. LSTMs were slightly more accurate, while the Transformer and Gated CNN ran considerably faster. Probing showed the same depth pattern in all models:
- the embedding layer captures only morphology;
- lower contextual layers capture local syntax;
- higher layers capture longer-range information such as pronominal coreference;
- the topmost layers specialize in next-token prediction.

This suggests biLMs act as general-purpose feature extractors regardless of architecture, so cheaper encoders are viable for transfer.

## Key points
- Perplexities on the 1 Billion Word Benchmark range from 37.5 (4-layer LSTM) to 44.5 (Gated CNN). The Transformer reaches 40.7, which the authors present as its first competitive LM result.
- The Transformer and Gated CNN contextual layers are about 3–5× faster than the 2-layer LSTM.
- Test-set comparison with GloVe:

  | Task (metric) | GloVe | 2-layer LSTM | Transformer | Gated CNN |
  |---|---|---|---|---|
  | MultiNLI matched (accuracy) | 77.0 | 79.6 | 79.4 | 78.3 |
  | SRL (F1) | 81.4 | 84.6 | 84.1 | 84.1 |
  | Constituency parsing (F1) | 91.8 | 93.9 | 93.7 | 92.9 |
  | NER (F1) | 89.9 | 91.7 | 91.1 | 91.2 |

- A linear classifier that labels spans from biLM layer activations alone reaches nearly 80% constituency F1. The same setup with GloVe gets only 18.1%.
- The learned ELMo layer weights peak below the top layer. The middle layers transfer best, while the top layers specialize in language modeling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

159 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 143 not yet.

- [FLAIR: An Easy-to-Use Framework for State-of-the-Art NLP](https://md.research.elamur.ai/free/en/papers/akbik-2019-flair-easy-use-framework-state/)
- [Analysis Methods in Neural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/)
- [Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation](https://md.research.elamur.ai/free/en/papers/geiger-2020-neural-natural-language-inference-models/)
- [A Structural Probe for Finding Syntax in Word Representations](https://md.research.elamur.ai/free/en/papers/hewitt-2019-structural-probe-finding-syntax-word/)
- [What Does BERT Learn about the Structure of Language?](https://md.research.elamur.ai/free/en/papers/jawahar-2019-what-does-bert-learn-about/)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://md.research.elamur.ai/free/en/papers/liu-2019-linguistic-knowledge-transferability-contextual-representations/)
- [S2ORC: The Semantic Scholar Open Research Corpus](https://md.research.elamur.ai/free/en/papers/lo-2019-s2orc-semantic-scholar-open-research/)
- [Language Models as Knowledge Bases?](https://md.research.elamur.ai/free/en/papers/petroni-2019-language-models-as-knowledge-bases/)
- [Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks](https://md.research.elamur.ai/free/en/papers/phang-2018-sentence-encoders-stilts-supplementary-training/)
- [Information-Theoretic Probing for Linguistic Structure](https://md.research.elamur.ai/free/en/papers/pimentel-2020-information-theoretic-probing-linguistic-structure/)
- [How Multilingual is Multilingual BERT?](https://md.research.elamur.ai/free/en/papers/pires-2019-how-multilingual-is-multilingual-bert/)
- [Transfer Learning in Natural Language Processing](https://md.research.elamur.ai/free/en/papers/ruder-2019-transfer-learning-natural-language-processing/)
- [oLMpics -- On what Language Model Pre-training Captures](https://md.research.elamur.ai/free/en/papers/talmor-2019-olmpics-language-model-pre-training/)
- [BERT Rediscovers the Classical NLP Pipeline](https://md.research.elamur.ai/free/en/papers/tenney-2019-bert-rediscovers-classical-nlp-pipeline/)
- [What do you learn from context? Probing for sentence structure in contextualized word representations](https://md.research.elamur.ai/free/en/papers/tenney-2019-learn-context-probing-sentence-structure/)
- [Explainability for Large Language Models: A Survey](https://md.research.elamur.ai/free/en/papers/zhao-2023-explainability-large-language-models-survey/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{peters2018dissecting,
  title = {{Dissecting Contextual Word Embeddings: Architecture and Representation}},
  author = {Peters, Matthew E. and Neumann, Mark and Zettlemoyer, Luke and Yih, Wen-tau},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1179},
  eprint = {1808.08949},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1179}
}
```

- [← previous Discourse Coherence: Concurrent Explicit and Implicit Relations](https://md.research.elamur.ai/free/en/papers/rohde-2018-discourse-coherence-concurrent-explicit-implicit/)
- [next → Document-Level Neural Machine Translation with Hierarchical Attention Networks](https://md.research.elamur.ai/free/en/papers/miculicich-2018-document-level-neural-machine-translation/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

52 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James R. Glass. 2017. What do neural machine translation models learn about morphology? In ACL. [find](https://scholar.google.com/scholar?q=Yonatan+Belinkov%2C+Nadir+Durrani%2C+Fahim+Dalvi%2C+Hassan+Sajjad%2C+and+James+R%2E+Glass%2E+2017%2E+What+do+neural+machine+translation+models+learn+about+morphology%3F+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/)
2. T. Blevins, O. Levy, and L. Zettlemoyer. 2018. Deep RNNs Encode Soft Hierarchical Syntax. In ACL. [find](https://scholar.google.com/scholar?q=T%2E+Blevins%2C+O%2E+Levy%2C+and+L%2E+Zettlemoyer%2E+2018%2E+Deep+RNNs+Encode+Soft+Hierarchical+Syntax%2E+In+ACL%2E)
3. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Ciprian+Chelba%2C+Tomas+Mikolov%2C+Mike+Schuster%2C+Qi+Ge%2C+Thorsten+Brants%2C+Phillipp+Koehn%2C+and+Tony+Robinson%2E+2014%2E+One+billion+word+benchmark+for+measuring+progress+in+statistical+language+modeling%2E+In+IN) [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
4. Qian Chen, Xiao-Dan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In ACL. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiao%2DDan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+lstm+for+natural+language+inference%2E+In+ACL%2E)
5. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single vector: Probing sentence embeddings for linguistic properties. In ACL. [find](https://scholar.google.com/scholar?q=Alexis+Conneau%2C+German+Kruszewski%2C+Guillaume+Lample%2C+Lo%C4%B1%CC%88c+Barrault%2C+and+Marco+Baroni%2E+2018%2E+What+you+can+cram+into+a+single+vector%3A+Probing+sentence+embeddings+for+linguistic+properties%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/conneau-2018-what-you-can-cram-into/)
6. Yann Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In ICML. [find](https://scholar.google.com/scholar?q=Yann+Dauphin%2C+Angela+Fan%2C+Michael+Auli%2C+and+David+Grangier%2E+2017%2E+Language+modeling+with+gated+convolutional+networks%2E+In+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/dauphin-2016-language-modeling-gated-convolutional-networks/)
7. David Gaddy, Mitchell Stern, and Dan Klein. 2018. What’s going on in neural constituency parsers? An analysis. In NAACL. [find](https://scholar.google.com/scholar?q=David+Gaddy%2C+Mitchell+Stern%2C+and+Dan+Klein%2E+2018%2E+What%E2%80%99s+going+on+in+neural+constituency+parsers%3F+An+analysis%2E+In+NAACL%2E)
8. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In ACL workshop for NLP Open Source Software. [find](https://scholar.google.com/scholar?q=Matt+Gardner%2C+Joel+Grus%2C+Mark+Neumann%2C+Oyvind+Tafjord%2C+Pradeep+Dasigi%2C+Nelson+F%2E+Liu%2C+Matthew+Peters%2C+Michael+Schmitz%2C+and+Luke+S%2E+Zettlemoyer%2E+2018%2E+AllenNLP%3A+A+deep+semantic+natural+language+process) [our card](https://md.research.elamur.ai/free/en/papers/gardner-2018-allennlp-deep-semantic-natural-language/)
9. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. 2017. Convolutional sequence to sequence learning. In ICML. [find](https://scholar.google.com/scholar?q=Jonas+Gehring%2C+Michael+Auli%2C+David+Grangier%2C+Denis+Yarats%2C+and+Yann+Dauphin%2E+2017%2E+Convolutional+sequence+to+sequence+learning%2E+In+ICML%2E) [our card](https://md.research.elamur.ai/free/en/papers/gehring-2017-convolutional-sequence-sequence-learning/)
10. Yichen Gong, Heng Luo, and Jian Zhang. 2018. Natural language inference over interaction space. In ICLR. [find](https://scholar.google.com/scholar?q=Yichen+Gong%2C+Heng+Luo%2C+and+Jian+Zhang%2E+2018%2E+Natural+language+inference+over+interaction+space%2E+In+ICLR%2E)
11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR. [find](https://scholar.google.com/scholar?q=Kaiming+He%2C+Xiangyu+Zhang%2C+Shaoqing+Ren%2C+and+Jian+Sun%2E+2016%2E+Deep+residual+learning+for+image+recognition%2E+In+CVPR%2E)
12. Luheng He, Kenton Lee, Mike Lewis, and Luke S. Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In ACL. [find](https://scholar.google.com/scholar?q=Luheng+He%2C+Kenton+Lee%2C+Mike+Lewis%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+Deep+semantic+role+labeling%3A+What+works+and+what%E2%80%99s+next%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/he-2017-deep-semantic-role-labeling-what/)
13. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. [find](https://scholar.google.com/scholar?q=Jeremy+Howard+and+Sebastian+Ruder%2E+2018%2E+Universal+language+model+fine%2Dtuning+for+text+classification%2E+In+ACL%2E)
14. Ákos Kádár, Grzegorz Chrupala, and Afra Alishahi. 2017. Representation of linguistic form and function in recurrent neural networks. Computational Linguistics, 43:761–780. [find](https://scholar.google.com/scholar?q=A%CC%81kos+Ka%CC%81da%CC%81r%2C+Grzegorz+Chrupala%2C+and+Afra+Alishahi%2E+2017%2E+Representation+of+linguistic+form+and+function+in+recurrent+neural+networks%2E+Computational+Linguistics%2C+43%3A761%E2%80%93780%2E)
15. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2015. Visualizing and understanding recurrent networks. In ICLR workshop. [find](https://scholar.google.com/scholar?q=Andrej+Karpathy%2C+Justin+Johnson%2C+and+Li+Fei%2DFei%2E+2015%2E+Visualizing+and+understanding+recurrent+networks%2E+In+ICLR+workshop%2E) [our card](https://md.research.elamur.ai/free/en/papers/karpathy-2015-visualizing-understanding-recurrent-networks/)
16. Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In ACL. [find](https://scholar.google.com/scholar?q=Urvashi+Khandelwal%2C+He+He%2C+Peng+Qi%2C+and+Dan+Jurafsky%2E+2018%2E+Sharp+nearby%2C+fuzzy+far+away%3A+How+neural+language+models+use+context%2E+In+ACL%2E)
17. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. In AAAI 2016. [find](https://scholar.google.com/scholar?q=Yoon+Kim%2C+Yacine+Jernite%2C+David+Sontag%2C+and+Alexander+M+Rush%2E+2015%2E+Character%2Daware+neural+language+models%2E+In+AAAI+2016%2E) [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
18. Nikita Kitaev and Dan Klein. 2018. Constituency parsing with a self-attentive encoder. In ACL. [find](https://scholar.google.com/scholar?q=Nikita+Kitaev+and+Dan+Klein%2E+2018%2E+Constituency+parsing+with+a+self%2Dattentive+encoder%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/kitaev-2018-constituency-parsing-self-attentive-encoder/)
19. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-NMT: Open-source toolkit for neural machine translation. In ACL. [find](https://scholar.google.com/scholar?q=Guillaume+Klein%2C+Yoon+Kim%2C+Yuntian+Deng%2C+Jean+Senellart%2C+and+Alexander+M%2E+Rush%2E+2017%2E+Open%2DNMT%3A+Open%2Dsource+toolkit+for+neural+machine+translation%2E+In+ACL%2E)
20. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In EACL. [find](https://scholar.google.com/scholar?q=Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+Lingpeng+Kong%2C+Chris+Dyer%2C+Graham+Neubig%2C+and+Noah+A%2E+Smith%2E+2017%2E+What+do+recurrent+neural+network+grammars+learn+about+syntax%3F+In+EACL%2E)
21. John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. [find](https://scholar.google.com/scholar?q=John+D%2E+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+ICML%2E)
22. Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP. [find](https://scholar.google.com/scholar?q=Kenton+Lee%2C+Luheng+He%2C+Mike+Lewis%2C+and+Luke+S%2E+Zettlemoyer%2E+2017%2E+End%2Dto%2Dend+neural+coreference+resolution%2E+In+EMNLP%2E) [our card](https://md.research.elamur.ai/free/en/papers/lee-2017-end-end-neural-coreference-resolution/)
23. Jiwei Li, Will Monroe, and Daniel Jurafsky. 2016. Understanding neural networks through representation erasure. arXiv, abs/1612.08220. — [arXiv:1612.08220](https://arxiv.org/abs/1612.08220) [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Will+Monroe%2C+and+Daniel+Jurafsky%2E+2016%2E+Understanding+neural+networks+through+representation+erasure%2E+arXiv%2C+abs%2F1612%2E08220%2E+%E2%80%94+%5BarXiv%3A1612%2E08220%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1612%2E08220%29)
24. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. In TACL. [find](https://scholar.google.com/scholar?q=Tal+Linzen%2C+Emmanuel+Dupoux%2C+and+Yoav+Goldberg%2E+2016%2E+Assessing+the+ability+of+LSTMs+to+learn+syntax%2Dsensitive+dependencies%2E+In+TACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/linzen-2016-assessing-ability-lstms-learn-syntax/)
25. Liyuan Liu, Xiang Ren, Jingbo Shang, Jian Peng, and Jiawei Han. 2018. Efficient contextualized representation: Language model pruning for sequence labeling. In EMNLP. [find](https://scholar.google.com/scholar?q=Liyuan+Liu%2C+Xiang+Ren%2C+Jingbo+Shang%2C+Jian+Peng%2C+and+Jiawei+Han%2E+2018%2E+Efficient+contextualized+representation%3A+Language+model+pruning+for+sequence+labeling%2E+In+EMNLP%2E)
26. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579–2605. [find](https://scholar.google.com/scholar?q=Laurens+van+der+Maaten+and+Geoffrey+Hinton%2E+2008%2E+Visualizing+data+using+t%2Dsne%2E+Journal+of+machine+learning+research%2C+9%28Nov%29%3A2579%E2%80%932605%2E)
27. Vidur Joshi, Matthew Peters, and Mark Hopkins. 2018. Extending a parser to distant domains using a few dozen partially annotated examples. In ACL. [find](https://scholar.google.com/scholar?q=Vidur+Joshi%2C+Matthew+Peters%2C+and+Mark+Hopkins%2E+2018%2E+Extending+a+parser+to+distant+domains+using+a+few+dozen+partially+annotated+examples%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/acl-2018-joshi-extending-parser-distant-domains-few/)
28. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Beatrice+Santorini%2C+and+Mary+Ann+Marcinkiewicz%2E+1993%2E+Building+a+Large+Annotated+Corpus+of+English%3A+The+Penn+Treebank%2E+Computational+Linguistics%2C+19%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
29. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv, abs/1602.02410. [find](https://scholar.google.com/scholar?q=Rafal+Jo%CC%81zefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+arXiv%2C+abs%2F1602%2E02410%2E)
30. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS 2017. [find](https://scholar.google.com/scholar?q=Bryan+McCann%2C+James+Bradbury%2C+Caiming+Xiong%2C+and+Richard+Socher%2E+2017%2E+Learned+in+translation%3A+Contextualized+word+vectors%2E+In+NIPS+2017%2E) [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
31. Gábor Melis, Chris Dyer, and Phil Blunsom. 2018. On the state of the art of evaluation in neural language models. In ICLR. [find](https://scholar.google.com/scholar?q=Ga%CC%81bor+Melis%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2018%2E+On+the+state+of+the+art+of+evaluation+in+neural+language+models%2E+In+ICLR%2E)
32. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In NIPS. [find](https://scholar.google.com/scholar?q=Rupesh+Kumar+Srivastava%2C+Klaus+Greff%2C+and+Ju%CC%88rgen+Schmidhuber%2E+2015%2E+Training+very+deep+networks%2E+In+NIPS%2E)
33. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2018. Regularizing and optimizing lstm language models. In ICLR. [find](https://scholar.google.com/scholar?q=Stephen+Merity%2C+Nitish+Shirish+Keskar%2C+and+Richard+Socher%2E+2018%2E+Regularizing+and+optimizing+lstm+language+models%2E+In+ICLR%2E) [our card](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/)
34. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In ICML. [find](https://scholar.google.com/scholar?q=Mukund+Sundararajan%2C+Ankur+Taly%2C+and+Qiqi+Yan%2E+2017%2E+Axiomatic+attribution+for+deep+networks%2E+In+ICML%2E)
35. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Efficient+estimation+of+word+representations+in+vector+space%2E+arXiv+preprint+arXiv%3A1301%2E3781%2E)
36. Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. In AAAI. [find](https://scholar.google.com/scholar?q=Zhixing+Tan%2C+Mingxuan+Wang%2C+Jun+Xie%2C+Yidong+Chen%2C+and+Xiaodong+Shi%2E+2018%2E+Deep+semantic+role+labeling+with+self%2Dattention%2E+In+AAAI%2E)
37. W. James Murdoch, Peter J. Liu, and Bin Yu. 2018. Beyond word importance: Contextual decomposition to extract interactions from lstms. In ICLR. [find](https://scholar.google.com/scholar?q=W%2E+James+Murdoch%2C+Peter+J%2E+Liu%2C+and+Bin+Yu%2E+2018%2E+Beyond+word+importance%3A+Contextual+decomposition+to+extract+interactions+from+lstms%2E+In+ICLR%2E)
38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. [find](https://scholar.google.com/scholar?q=Ashish+Vaswani%2C+Noam+Shazeer%2C+Niki+Parmar%2C+Jakob+Uszkoreit%2C+Llion+Jones%2C+Aidan+N%2E+Gomez%2C+Lukasz+Kaiser%2C+and+Illia+Polosukhin%2E+2017%2E+Attention+is+all+you+need%2E+In+NIPS%2E)
39. Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016. Wavenet: A generative model for raw audio. In arXiv, volume abs/1609.03499. [find](https://scholar.google.com/scholar?q=Aa%CC%88ron+van+den+Oord%2C+Sander+Dieleman%2C+Heiga+Zen%2C+Karen+Simonyan%2C+Oriol+Vinyals%2C+Alex+Graves%2C+Nal+Kalchbrenner%2C+Andrew+W%2E+Senior%2C+and+Koray+Kavukcuoglu%2E+2016%2E+Wavenet%3A+A+generative+model+for+raw+audio%2E) [our card](https://md.research.elamur.ai/free/en/papers/oord-2016-wavenet-generative-model-raw-audio/)
40. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL. [find](https://scholar.google.com/scholar?q=Adina+Williams%2C+Nikita+Nangia%2C+and+Samuel+Bowman%2E+2018%2E+A+broad%2Dcoverage+challenge+corpus+for+sentence+understanding+through+inference%2E+In+NAACL%2E)
41. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+EMNLP%2E)
42. Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision, pages 818– 833. [find](https://scholar.google.com/scholar?q=Matthew+D+Zeiler+and+Rob+Fergus%2E+2014%2E+Visualizing+and+understanding+convolutional+networks%2E+In+European+conference+on+computer+vision%2C+pages+818%E2%80%93+833%2E) [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2013-visualizing-understanding-convolutional-networks/)
43. Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Waleed+Ammar%2C+Chandra+Bhagavatula%2C+and+Russell+Power%2E+2017%2E+Semi%2Dsupervised+sequence+tagging+with+bidirectional+language+models%2E+In+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/)
44. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL. [find](https://scholar.google.com/scholar?q=Matthew+E%2E+Peters%2C+Mark+Neumann%2C+Mohit+Iyyer%2C+Matt+Gardner%2C+Christopher+Clark%2C+Kenton+Lee%2C+and+Luke+Zettlemoyer%2E+2018%2E+Deep+contextualized+word+representations%2E+In+NAACL%2E)
45. Matthew D. Zeiler. 2012. ADADELTA: An adaptive learning rate method. arXiv, abs/1212.5701. [find](https://scholar.google.com/scholar?q=Matthew+D%2E+Zeiler%2E+2012%2E+ADADELTA%3A+An+adaptive+learning+rate+method%2E+arXiv%2C+abs%2F1212%2E5701%2E) [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)
46. Kelly Zhang and Samuel R. Bowman. 2018. Language modeling teaches you more syntax than translation does: Lessons learned through auxiliary task analysis. arXiv preprint. [find](https://scholar.google.com/scholar?q=Kelly+Zhang+and+Samuel+R%2E+Bowman%2E+2018%2E+Language+modeling+teaches+you+more+syntax+than+translation+does%3A+Lessons+learned+through+auxiliary+task+analysis%2E+arXiv+preprint%2E)
47. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In CoNLL. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Hwee+Tou+Ng%2C+Anders+Bjo%CC%88rkelund%2C+Olga+Uryupina%2C+Yuchen+Zhang%2C+and+Zhi+Zhong%2E+2013%2E+Towards+robust+linguistic+analysis+using+ontonotes%2E+In+CoNLL%2E)
48. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In EMNLP-CoNLL Shared Task. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Olga+Uryupina%2C+and+Yuchen+Zhang%2E+2012%2E+Conll%2D2012+shared+task%3A+Modeling+multilingual+unrestricted+coreference+in+ontonotes%2E+In+EMNLP%2DCoNLL+Shared+Tas) [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
49. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. [find](https://scholar.google.com/scholar?q=Alec+Radford%2C+Karthik+Narasimhan%2C+Tim+Salimans%2C+and+Ilya+Sutskever%2E+2018%2E+Improving+language+understanding+by+generative+pre%2Dtraining%2E)
50. Hasim Sak, Andrew W. Senior, and Franoise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH. [find](https://scholar.google.com/scholar?q=Hasim+Sak%2C+Andrew+W%2E+Senior%2C+and+Franoise+Beaufays%2E+2014%2E+Long+short%2Dterm+memory+recurrent+neural+network+architectures+for+large+scale+acoustic+modeling%2E+In+INTERSPEECH%2E)
51. Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In CoNLL/LLL. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Sabine+Buchholz%2E+2000%2E+Introduction+to+the+CoNLL%2D2000+shared+task+chunking%2E+In+CoNLL%2FLLL%2E) [our card](https://md.research.elamur.ai/free/en/papers/sang-2000-introduction-conll-2000-shared-task/)
52. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In CoNLL. [find](https://scholar.google.com/scholar?q=Erik+F%2E+Tjong+Kim+Sang+and+Fien+De+Meulder%2E+2003%2E+Introduction+to+the+CoNLL%2D2003+shared+task%3A+Language%2Dindependent+named+entity+recognition%2E+In+CoNLL%2E)