---
title: "Deep Contextualized Word Representations — Peters 2018"
description: "The authors, from the Allen Institute for AI and the University of Washington, propose ELMo… 61 references, cited by 3,240 papers here."
authors: ["Matthew E. Peters", "Mark Neumann", "Mohit Iyyer", "Matt Gardner", "Christopher Clark", "Kenton Lee", "Luke Zettlemoyer"]
year: 2018
venue: "Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)"
source: "https://aclanthology.org/N18-1202/"
doi: "10.18653/v1/n18-1202"
arxiv: "1802.05365"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2018](https://md.research.elamur.ai/free/en/papers/2018/) / Deep Contextualized Word Representations

# Deep Contextualized Word Representations

**Authors:** [Matthew E. Peters](https://md.research.elamur.ai/free/en/authors/matthew-peters/), [Mark Neumann](https://md.research.elamur.ai/free/en/authors/mark-neumann/), [Mohit Iyyer](https://md.research.elamur.ai/free/en/authors/mohit-iyyer/), [Matt Gardner](https://md.research.elamur.ai/free/en/authors/matt-gardner/), [Christopher Clark](https://md.research.elamur.ai/free/en/authors/christopher-clark/), [Kenton Lee](https://md.research.elamur.ai/free/en/authors/kenton-lee/) and [Luke Zettlemoyer](https://md.research.elamur.ai/free/en/authors/luke-zettlemoyer/) · **Year:** 2018 · **Venue:** Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N18-1202/) · [PDF](https://aclanthology.org/N18-1202.pdf) · [DOI 10.18653/v1/n18-1202](https://doi.org/10.18653/v1/n18-1202) · [arXiv 1802.05365](https://arxiv.org/abs/1802.05365)

**Citations:** [3240](https://md.research.elamur.ai/free/en/papers/peters-2018-deep-contextualized-word-representations/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary

Classical word vector representations, such as word2vec or GloVe, assign each word a single vector regardless of context. As a result, they cannot distinguish the senses of polysemous words. The authors, from the Allen Institute for AI and the University of Washington, propose ELMo (Embeddings from Language Models). In this approach, the vector of each token is computed taking the entire sentence into account. The foundation is a two-layer bidirectional language model (biLM) built on LSTMs with character convolutions. Its pretraining used a single-language corpus (Chelba et al., 2014) containing about 30 million sentences. Previous approaches (TagLM, CoVe) used only the top layer of the network. ELMo, by contrast, learns a task-specific weighted combination of all biLM layers for each target task. The weights of the biLM itself remain frozen, and the resulting vector is added to the input, and sometimes also to the output, of an existing model. This simple addition raised the best known results on six tasks, among them sentiment analysis, textual entailment, and question answering. The relative error reduction reached 20%. The paper showed that contextual representations from a large language model transfer easily across tasks. It is one of the key steps toward modern pretrained language models.

## Key points

- **Different layers encode different information.** Probing with a linear classifier on POS tagging (the WSJ portion of the Penn Treebank) showed that syntax is better captured by the lower biLM layer. The contextual meaning of a word for word sense disambiguation is better captured by the top layer. In both tests, the biLM outperforms the CoVe encoder.
- **Training efficiency.** The SRL model without ELMo reaches its best dev F1 only after 486 epochs. With ELMo, it surpasses this level as early as epoch 10, i.e., approximately 98% fewer updates are needed. With 1% of the training data, the model with ELMo achieves roughly the same F1 as the baseline model with 10%.
- **Ablation (Table 7, dev).** F1 on SQuAD: GloVe only — 80.8, ELMo character layer only — 81.4, full ELMo — 85.3, ELMo + GloVe — 85.6. Thus, the gain comes primarily from contextual information rather than subword information. Adding GloVe to ELMo yields only a marginal improvement (SRL: 84.5 → 84.7).
- **All layers are better than the top layer.** A combination of all biLM layers works better than the top LSTM layer alone. This holds for both ELMo and CoVe. Where a direct comparison is possible, ELMo outperforms CoVe, which builds its representations on top of a machine translation encoder.
- **Open resource.** The trained models and code are published at allennlp.org/elmo.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/)

## Cite

Cite as: Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{peters2018deep,
  title = {{Deep Contextualized Word Representations}},
  author = {Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee and Luke Zettlemoyer},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1202},
  eprint = {1802.05365},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n18-1202}
}
```

- [← previous Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/)
- [next → Continual Lifelong Learning with Neural Networks: A Review](https://md.research.elamur.ai/free/en/papers/parisi-2018-continual-lifelong-learning-neural-networks/)
- [↑ back to papers of 2018](https://md.research.elamur.ai/free/en/papers/2018/)

## References

61 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR abs/1607.06450. [our card](https://md.research.elamur.ai/free/en/papers/ba-2016-layer-normalization/)
2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James R. Glass. 2017. What do neural machine translation models learn about morphology? In ACL. [our card](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/)
3. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL 5:135–146. [our card](https://md.research.elamur.ai/free/en/papers/bojanowski-2017-enriching-word-vectors-subword-information/)
4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
5. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling. In INTERSPEECH. [our card](https://md.research.elamur.ai/free/en/papers/chelba-2013-one-billion-word-benchmark-measuring/)
6. Qian Chen, Xiao-Dan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In ACL. [find](https://scholar.google.com/scholar?q=Qian+Chen%2C+Xiao%2DDan+Zhu%2C+Zhen%2DHua+Ling%2C+Si+Wei%2C+Hui+Jiang%2C+and+Diana+Inkpen%2E+2017%2E+Enhanced+lstm+for+natural+language+inference%2E+In+ACL%2E)
7. Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. In TACL. [our card](https://md.research.elamur.ai/free/en/papers/chiu-2015-named-entity-recognition-bidirectional-lstm/)
8. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In SSST@EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/)
9. Christopher Clark and Matthew Gardner. 2017. Simple and effective multi-paragraph reading comprehension. CoRR abs/1710.10723. [our card](https://md.research.elamur.ai/free/en/papers/clark-2017-simple-effective-multi-paragraph-reading/)
10. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/clark-2016-deep-reinforcement-learning-mention-ranking/)
11. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. Natural language processing (almost) from scratch. In JMLR. [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/)
12. Andrew M. Dai and Quoc V. Le. 2015. supervised sequence learning. In NIPS. [find](https://scholar.google.com/scholar?q=Andrew+M%2E+Dai+and+Quoc+V%2E+Le%2E+2015%2E+supervised+sequence+learning%2E+In+NIPS%2E)
13. Semi-Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/durrett-2013-easy-victories-uphill-battles-coreference/)
14. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In NIPS. [our card](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/)
15. Yichen Gong, Heng Luo, and Jian Zhang. 2018. Natural language inference over interaction space. In ICLR. [find](https://scholar.google.com/scholar?q=Yichen+Gong%2C+Heng+Luo%2C+and+Jian+Zhang%2E+2018%2E+Natural+language+inference+over+interaction+space%2E+In+ICLR%2E)
16. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple nlp tasks. In EMNLP 2017. [our card](https://md.research.elamur.ai/free/en/papers/hashimoto-2016-joint-many-task-model-growing/)
17. Luheng He, Kenton Lee, Mike Lewis, and Luke S. Zettlemoyer. 2017. Deep semantic role labeling: What works and what’s next. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/he-2017-deep-semantic-role-labeling-what/)
18. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+Computation+9%2E)
19. Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for word sense disambiguation: An evaluation study. In ACL. [find](https://scholar.google.com/scholar?q=Ignacio+Iacobacci%2C+Mohammad+Taher+Pilehvar%2C+and+Roberto+Navigli%2E+2016%2E+Embeddings+for+word+sense+disambiguation%3A+An+evaluation+study%2E+In+ACL%2E)
20. Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2017. Stochastic answer networks for machine reading comprehension. arXiv preprint arXiv:1712.03556 . [find](https://scholar.google.com/scholar?q=Xiaodong+Liu%2C+Yelong+Shen%2C+Kevin+Duh%2C+and+Jianfeng+Gao%2E+2017%2E+Stochastic+answer+networks+for+machine+reading+comprehension%2E+arXiv+preprint+arXiv%3A1712%2E03556+%2E)
21. Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/ma-2016-end-end-sequence-labeling-bi/)
22. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics 19:313–330. [our card](https://md.research.elamur.ai/free/en/papers/marcus-1993-building-large-annotated-corpus-english/)
23. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In NIPS 2017. [our card](https://md.research.elamur.ai/free/en/papers/mccann-2017-learned-translation-contextualized-word-vectors/)
24. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional lstm. In CoNLL. [our card](https://md.research.elamur.ai/free/en/papers/melamud-2016-context2vec-learning-generic-context-embedding/)
25. Gábor Melis, Chris Dyer, and Phil Blunsom. 2017. On the state of the art of evaluation in neural language models. CoRR abs/1707.05589. [find](https://scholar.google.com/scholar?q=Ga%CC%81bor+Melis%2C+Chris+Dyer%2C+and+Phil+Blunsom%2E+2017%2E+On+the+state+of+the+art+of+evaluation+in+neural+language+models%2E+CoRR+abs%2F1707%2E05589%2E)
26. Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. CoRR abs/1602.02410. [find](https://scholar.google.com/scholar?q=Rafal+Jo%CC%81zefowicz%2C+Oriol+Vinyals%2C+Mike+Schuster%2C+Noam+Shazeer%2C+and+Yonghui+Wu%2E+2016%2E+Exploring+the+limits+of+language+modeling%2E+CoRR+abs%2F1602%2E02410%2E)
27. Stephen Merity, Nitish Shirish Keskar, and Richard Socher. 2017. Regularizing and optimizing lstm language models. CoRR abs/1708.02182. [our card](https://md.research.elamur.ai/free/en/papers/merity-2017-regularizing-optimizing-lstm-language-models/)
28. Rafal Józefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In ICML. [find](https://scholar.google.com/scholar?q=Rafal+Jo%CC%81zefowicz%2C+Wojciech+Zaremba%2C+and+Ilya+Sutskever%2E+2015%2E+An+empirical+exploration+of+recurrent+network+architectures%2E+In+ICML%2E)
29. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In NIPS. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
30. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware neural language models. In AAAI 2016. [our card](https://md.research.elamur.ai/free/en/papers/kim-2015-character-aware-neural-language-models/)
31. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2015%2E+Adam%3A+A+method+for+stochastic+optimization%2E+In+ICLR%2E)
32. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, Ishaan Gulrajani James Bradbury, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In ICML. [our card](https://md.research.elamur.ai/free/en/papers/kumar-2015-ask-me-anything-dynamic-memory/)
33. John D. Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML. [find](https://scholar.google.com/scholar?q=John+D%2E+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E+In+ICML%2E)
34. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL-HLT. [our card](https://md.research.elamur.ai/free/en/papers/lample-2016-neural-architectures-named-entity-recognition/)
35. Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/lee-2017-end-end-neural-coreference-resolution/)
36. Wang Ling, Chris Dyer, Alan W. Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luı́s Marujo, and Tiago Luı́s. 2015. Finding function in form: Compositional character models for open vocabulary word representation. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/ling-2015-finding-function-form-compositional-character/)
37. George A. Miller, Martin Chodorow, Shari Landes, Claudia Leacock, and Robert G. Thomas. 1994. Using a semantic concordance for sense identification. In HLT. [our card](https://md.research.elamur.ai/free/en/papers/miller-1994-semantic-concordance-sense-identification/)
38. Tsendsuren Munkhdalai and Hong Yu. 2017. Neural tree indexers for text understanding. In EACL. [find](https://scholar.google.com/scholar?q=Tsendsuren+Munkhdalai+and+Hong+Yu%2E+2017%2E+Neural+tree+indexers+for+text+understanding%2E+In+EACL%2E)
39. Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, and Andrew McCallum. 2014. Efficient non-parametric estimation of multiple embeddings per word in vector space. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/neelakantan-2014-efficient-non-parametric-estimation-multiple/)
40. Martha Palmer, Paul Kingsbury, and Daniel Gildea. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics 31:71– 106. [our card](https://md.research.elamur.ai/free/en/papers/palmer-2005-proposition-bank-annotated-corpus-semantic/)
41. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/pennington-2014-glove-global-vectors-word-representation/)
42. Matthew E. Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. 2017. Semi-supervised sequence tagging with bidirectional language models. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/peters-2017-semi-supervised-sequence-tagging-bidirectional/)
43. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In CoNLL. [find](https://scholar.google.com/scholar?q=Sameer+Pradhan%2C+Alessandro+Moschitti%2C+Nianwen+Xue%2C+Hwee+Tou+Ng%2C+Anders+Bjo%CC%88rkelund%2C+Olga+Uryupina%2C+Yuchen+Zhang%2C+and+Zhi+Zhong%2E+2013%2E+Towards+robust+linguistic+analysis+using+ontonotes%2E+In+CoNLL%2E)
44. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In EMNLP-CoNLL Shared Task. [our card](https://md.research.elamur.ai/free/en/papers/pradhan-2012-conll-2012-shared-task-modeling/)
45. Alessandro Raganato, Claudio Delli Bovi, and Roberto Navigli. 2017a. Neural sequence learning models for word sense disambiguation. In EMNLP. [find](https://scholar.google.com/scholar?q=Alessandro+Raganato%2C+Claudio+Delli+Bovi%2C+and+Roberto+Navigli%2E+2017a%2E+Neural+sequence+learning+models+for+word+sense+disambiguation%2E+In+EMNLP%2E)
46. Alessandro Raganato, Jose Camacho-Collados, and Roberto Navigli. 2017b. Word sense disambiguation: A unified evaluation framework and empirical comparison. In EACL. [our card](https://md.research.elamur.ai/free/en/papers/raganato-2017-word-sense-disambiguation-unified-evaluation/)
47. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In HLT-NAACL. [our card](https://md.research.elamur.ai/free/en/papers/wiseman-2016-learning-global-features-coreference-resolution/)
48. Matthew D. Zeiler. 2012. Adadelta: An adaptive learning rate method. CoRR abs/1212.5701. [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)
49. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/zhou-2015-end-end-learning-semantic-role/)
50. Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text classification improved by integrating bidirectional lstm with two-dimensional max pooling. In COLING. [find](https://scholar.google.com/scholar?q=Peng+Zhou%2C+Zhenyu+Qi%2C+Suncong+Zheng%2C+Jiaming+Xu%2C+Hongyun+Bao%2C+and+Bo+Xu%2E+2016%2E+Text+classification+improved+by+integrating+bidirectional+lstm+with+two%2Ddimensional+max+pooling%2E+In+COLING%2E)
51. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP. [our card](https://md.research.elamur.ai/free/en/papers/rajpurkar-2016-squad-100-000-questions-machine/)
52. Prajit Ramachandran, Peter Liu, and Quoc Le. 2017. Improving sequence to sequence learning with unlabeled data. In EMNLP. [find](https://scholar.google.com/scholar?q=Prajit+Ramachandran%2C+Peter+Liu%2C+and+Quoc+Le%2E+2017%2E+Improving+sequence+to+sequence+learning+with+unlabeled+data%2E+In+EMNLP%2E)
53. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In CoNLL. [our card](https://md.research.elamur.ai/free/en/papers/sang-2003-introduction-conll-2003-shared-task/)
54. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. [our card](https://md.research.elamur.ai/free/en/papers/seo-2016-bidirectional-attention-flow-machine-comprehension/)
55. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. [find](https://scholar.google.com/scholar?q=Richard+Socher%2C+Alex+Perelygin%2C+Jean+Y+Wu%2C+Jason+Chuang%2C+Christopher+D+Manning%2C+Andrew+Y+Ng%2C+and+Christopher+Potts%2E+2013%2E+Recursive+deep+models+for+semantic+compositionality+over+a+sentiment+treebank%2E)
56. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In ACL 2016. [our card](https://md.research.elamur.ai/free/en/papers/sogaard-2016-deep-multi-task-learning-low/)
57. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research 15:1929–1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+E%2E+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+a+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+R)
58. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In NIPS. [find](https://scholar.google.com/scholar?q=Rupesh+Kumar+Srivastava%2C+Klaus+Greff%2C+and+Ju%CC%88rgen+Schmidhuber%2E+2015%2E+Training+very+deep+networks%2E+In+NIPS%2E)
59. Joseph P. Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/turian-2010-word-representations-simple-general-method/)
60. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In ACL. [our card](https://md.research.elamur.ai/free/en/papers/wang-2017-gated-self-matching-networks-reading/)
61. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Charagram: Embedding words and sentences via character n-grams. In EMNLP. [find](https://scholar.google.com/scholar?q=John+Wieting%2C+Mohit+Bansal%2C+Kevin+Gimpel%2C+and+Karen+Livescu%2E+2016%2E+Charagram%3A+Embedding+words+and+sentences+via+character+n%2Dgrams%2E+In+EMNLP%2E)