---
title: "Reporting Score Distributions Makes a — Reimers 2017"
description: "Neural sequence taggers train non-deterministically, so the random seed alone can push a model into local minima that generalize differently."
authors: ["Nils Reimers", "Iryna Gurevych"]
year: 2017
venue: "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D17-1035/"
doi: "10.18653/v1/d17-1035"
arxiv: "1707.09861"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/reimers-2017-reporting-score-distributions-makes-difference/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/reimers-2017-reporting-score-distributions-makes-difference/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

# Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging

**Authors:** [Nils Reimers](https://md.research.elamur.ai/free/en/authors/nils-reimers/), [Iryna Gurevych](https://md.research.elamur.ai/free/en/authors/iryna-gurevych/) · **Year:** 2017 · **Venue:** Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D17-1035/) · [PDF](https://aclanthology.org/D17-1035.pdf) · [DOI 10.18653/v1/d17-1035](https://doi.org/10.18653/v1/d17-1035) · [arXiv 1707.09861](https://arxiv.org/abs/1707.09861) · [OpenAlex](https://openalex.org/W2740215900)

**Citations:** 455 citations (openalex-snapshot · 2026-10-14) · [193](#cited-by) in our library, of the 140673 papers whose bibliographies we read

## Summary
Neural sequence taggers train non-deterministically, so the random seed alone can push a model into local minima that generalize differently. Reimers and Gurevych argue that comparing systems by one test score is therefore unreliable. They repeatedly retrained two recent NER systems, varying nothing but the seed: the one by Lample et al. (2016) and the one by Ma and Hovy (2016). They also trained about 50,000 BiLSTM taggers on five tasks: POS (Penn Treebank), Chunking (CoNLL 2000), NER (CoNLL 2003), Entities (ACE 2005) and Events (TempEval 3).

Seed choice alone produced statistically significant gaps of up to a full F1 point. Picking the best run on the development set did not fix this. The comparison of score distributions also reversed the published ranking of the two NER systems.

The authors propose reporting distributions over multiple runs instead of single scores. Using distributions, they identify design choices that give both higher and more stable performance. Their implementation is released on GitHub (UKPLab/emnlp2017-bilstm-cnn-crf). The work matters for fair benchmarking and reproducibility in NLP.

## Key points
- **Ma and Hovy system (86 seeds):** F1 ranged from 89.99% to 91.00%, against a reported 91.21%.
- **Lample et al. system (41 seeds):** F1 ranged from 90.19% to 91.14%, with a higher median and smaller variance. A Kolmogorov-Smirnov test showed the two distributions differ (p \< 0.01), so Lample et al. comes out ahead, reversing the published ranking.
- **Development scores are a weak guide:** dev and test scores correlated only weakly (Spearman ρ = 0.229). For Lample et al., the run with the best dev score reached only 90.31% on test.
- **Seed-only differences across tasks:** on CoNLL 2003 NER, two runs with identical hyperparameters differed by 0.38% F1 at the median and 1.08% at the 95th percentile. On ACE 2005 Entities, the gap reached 8.23 points (82.5% vs 74.3%).
- **Robust design choices:**
  - Variational dropout was the best option in 83.5% of 479 configurations.
  - Gradient normalization helped, while gradient clipping did not.
  - Adam, Nadam and RMSProp varied less than SGD, Adagrad and Adadelta.
  - Two stacked LSTM layers generally performed best.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/)

Topics: [Named entity recognition](https://md.research.elamur.ai/free/en/topics/named-entity-recognition/)

## Cited by

Cited by

193 papers of our library cite it (of the 140673 whose bibliographies we read); 23 of them have a page here, 170 not yet.

- [Contextual String Embeddings for Sequence Labeling](https://md.research.elamur.ai/free/en/papers/akbik-2018-contextual-string-embeddings-sequence-labeling/)
- [Pooled Contextualized Embeddings for Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/akbik-2019-pooled-contextualized-embeddings-named-entity/)
- [Leveraging Pretrained Word Embeddings for Part-of-Speech Tagging of Code Switching Data](https://md.research.elamur.ai/free/en/papers/alghamdi-2019-leveraging-pretrained-word-embeddings-part/)
- [Graph-to-Sequence Learning using Gated Graph Neural Networks](https://md.research.elamur.ai/free/en/papers/beck-2018-graph-sequence-learning-gated-graph/)
- [Semi-Supervised Sequence Modeling with Cross-View Training](https://md.research.elamur.ai/free/en/papers/clark-2018-semi-supervised-sequence-modeling-cross/)
- [Structural Scaffolds for Citation Intent Classification in Scientific Publications](https://md.research.elamur.ai/free/en/papers/cohan-2019-structural-scaffolds-citation-intent-classification/)
- [STANDER: An Expert-Annotated Dataset for News Stance Detection and Evidence Retrieval](https://md.research.elamur.ai/free/en/papers/conforti-2020-stander-expert-annotated-dataset-news/)
- [VerbAtlas: a Novel Large-Scale Verbal Semantic Resource and Its Application to Semantic Role Labeling](https://md.research.elamur.ai/free/en/papers/fabio-2019-verbatlas-novel-large-scale-verbal/)
- [MiqraBERT: Regression-Based Sentence-BERT Finetuning for Biblical Hebrew Parallel Detection](https://md.research.elamur.ai/free/en/papers/hbo-2026-smiley-miqrabert-regression-based-sentence-bert/)
- [BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languages](https://md.research.elamur.ai/free/en/papers/heinzerling-2018-bpemb-tokenization-free-pre-trained/)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://md.research.elamur.ai/free/en/papers/hou-2020-few-shot-slot-tagging-collapsed/)
- [Scheduled Multi-Task Learning: From Syntax to Translation](https://md.research.elamur.ai/free/en/papers/kiperwasser-2018-scheduled-multi-task-learning-syntax/)
- [Representing Social Media Users for Sarcasm Detection](https://md.research.elamur.ai/free/en/papers/kolchinski-2018-representing-social-media-users-sarcasm/)
- [Discourse Coherence in the Wild: A Dataset, Evaluation and Methods](https://md.research.elamur.ai/free/en/papers/lai-2018-discourse-coherence-wild-dataset-evaluation/)
- [Empower Sequence Labeling with Task-Aware Neural Language Model](https://md.research.elamur.ai/free/en/papers/liu-2017-empower-sequence-labeling-task-aware/)
- [Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation](https://md.research.elamur.ai/free/en/papers/reimers-2020-making-monolingual-sentence-embeddings-multilingual/)
- [Cross-topic Argument Mining from Heterogeneous Sources](https://md.research.elamur.ai/free/en/papers/stab-2018-cross-topic-argument-mining-heterogeneous/)
- [IMPLI: Investigating NLI Models' Performance on Figurative Language](https://md.research.elamur.ai/free/en/papers/stowe-2022-impli-investigating-nli-models-performance/)
- [Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks](https://md.research.elamur.ai/free/en/papers/thakur-2021-augmented-sbert-data-augmentation-method/)
- [Towards JointUD: Part-of-speech Tagging and Lemmatization using Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/ud-2018-arakelyan-jointud-part-speech-tagging-lemmatization/)
- [Parsing Tweets into Universal Dependencies](https://md.research.elamur.ai/free/en/papers/ud-2018-liu-parsing-tweets-into-universal-dependencies/)
- [Integrating Graph-Based and Transition-Based Dependency Parsers in the Deep Contextualized Era](https://md.research.elamur.ai/free/en/papers/ud-2020-falenska-integrating-graph-based-transition-based/)
- [Double Embeddings and CNN-based Sequence Labeling for Aspect Extraction](https://md.research.elamur.ai/free/en/papers/xu-2018-double-embeddings-cnn-based-sequence/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{reimers2017reporting,
  title = {{Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Tagging}},
  author = {Reimers, Nils and Gurevych, Iryna},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1035},
  eprint = {1707.09861},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1035}
}
```

- [← previous Regularizing Neural Networks by Penalizing Confident Output Distributions](https://md.research.elamur.ai/free/en/papers/pereyra-2017-regularizing-neural-networks-penalizing-confident/)
- [next → Results of the WMT17 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/)

## References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. CoRR, abs/1603.06042. [find](https://scholar.google.com/scholar?q=Daniel+Andor%2C+Chris+Alberti%2C+David+Weiss%2C+Aliaksei+Severyn%2C+Alessandro+Presta%2C+Kuzman+Ganchev%2C+Slav+Petrov%2C+and+Michael+Collins%2E+2016%2E+Globally+normalized+transition%2Dbased+neural+networks%2E+CoRR%2C+abs%2F1) [our card](https://md.research.elamur.ai/free/en/papers/andor-2016-globally-normalized-transition-based-neural/)
2. Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching Word Vectors with Subword Information. arXiv preprint arXiv:1607.04606. [find](https://scholar.google.com/scholar?q=Piotr+Bojanowski%2C+Edouard+Grave%2C+Armand+Joulin%2C+and+Tomas+Mikolov%2E+2016%2E+Enriching+Word+Vectors+with+Subword+Information%2E+arXiv+preprint+arXiv%3A1607%2E04606%2E)
3. Timothy Dozat. 2015. Incorporating Nesterov Momentum into Adam. [find](https://scholar.google.com/scholar?q=Timothy+Dozat%2E+2015%2E+Incorporating+Nesterov+Momentum+into+Adam%2E)
4. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization. J. Mach. Learn. Res., 12:2121–2159. [find](https://scholar.google.com/scholar?q=John+Duchi%2C+Elad+Hazan%2C+and+Yoram+Singer%2E+2011%2E+Adaptive+Subgradient+Methods+for+Online+Learning+and+Stochastic+Optimization%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+12%3A2121%E2%80%932159%2E)
5. Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, and Samy Bengio. 2010. Why Does Unsupervised Pre-training Help Deep Learning? Journal of Machine Learning Research, 11:625–660. [find](https://scholar.google.com/scholar?q=Dumitru+Erhan%2C+Yoshua+Bengio%2C+Aaron+Courville%2C+Pierre%2DAntoine+Manzagol%2C+Pascal+Vincent%2C+and+Samy+Bengio%2E+2010%2E+Why+Does+Unsupervised+Pre%2Dtraining+Help+Deep+Learning%3F+Journal+of+Machine+Learning+Resear)
6. Antske Fokkens, Marieke van Erp, Marten Postma, Ted Pedersen, Piek Vossen, and Nuno Freire. 2013. Offspring from Reproduction Problems: What Replication Failure Teaches Us. In ACL (1), pages 1691–1701. The Association for Computer Linguistics. [find](https://scholar.google.com/scholar?q=Antske+Fokkens%2C+Marieke+van+Erp%2C+Marten+Postma%2C+Ted+Pedersen%2C+Piek+Vossen%2C+and+Nuno+Freire%2E+2013%2E+Offspring+from+Reproduction+Problems%3A+What+Replication+Failure+Teaches+Us%2E+In+ACL+%281%29%2C+pages+1691%E2%80%931701)
7. Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1019–1027. [find](https://scholar.google.com/scholar?q=Yarin+Gal+and+Zoubin+Ghahramani%2E+2016%2E+A+Theoretically+Grounded+Application+of+Dropout+in+Recurrent+Neural+Networks%2E+In+Advances+in+Neural+Information+Processing+Systems+29%3A+Annual+Conference+on+Neura) [our card](https://md.research.elamur.ai/free/en/papers/gal-2015-theoretically-grounded-application-dropout-recurrent/)
8. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. CoRR, abs/1609.04836. [find](https://scholar.google.com/scholar?q=Nitish+Shirish+Keskar%2C+Dheevatsa+Mudigere%2C+Jorge+Nocedal%2C+Mikhail+Smelyanskiy%2C+and+Ping+Tak+Peter+Tang%2E+2016%2E+On+Large%2DBatch+Training+for+Deep+Learning%3A+Generalization+Gap+and+Sharp+Minima%2E+CoRR%2C+abs%2F)
9. Diederik P. Kingma and Jimmy Ba. 2014. A Method for Stochastic Optimization. abs/1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P%2E+Kingma+and+Jimmy+Ba%2E+2014%2E+A+Method+for+Stochastic+Optimization%2E+abs%2F1412%2E6980%2E)
10. Alexandros Komninos and Suresh Manandhar. 2016. Dependency based embeddings for sentence classification tasks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1490–1500, San Diego, California. As… [find](https://scholar.google.com/scholar?q=Alexandros+Komninos+and+Suresh+Manandhar%2E+2016%2E+Dependency+based+embeddings+for+sentence+classification+tasks%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+fo)
11. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. CoRR, abs/1603.01360. [find](https://scholar.google.com/scholar?q=Guillaume+Lample%2C+Miguel+Ballesteros%2C+Sandeep+Subramanian%2C+Kazuya+Kawakami%2C+and+Chris+Dyer%2E+2016%2E+Neural+architectures+for+named+entity+recognition%2E+CoRR%2C+abs%2F1603%2E01360%2E)
12. Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4):541–551. [find](https://scholar.google.com/scholar?q=Y%2E+LeCun%2C+B%2E+Boser%2C+J%2E+S%2E+Denker%2C+D%2E+Henderson%2C+R%2E+E%2E+Howard%2C+W%2E+Hubbard%2C+and+L%2E+D%2E+Jackel%2E+1989%2E+Backpropagation+Applied+to+Handwritten+Zip+Code+Recognition%2E+Neural+Computation%2C+1%284%29%3A541%E2%80%93551%2E)
13. Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. 1998. Efficient BackProp. In Neural Networks: Tricks of the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, pages 9–50, London, UK, UK. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Yann+LeCun%2C+Le%CC%81on+Bottou%2C+Genevieve+B%2E+Orr%2C+and+Klaus%2DRobert+Mu%CC%88ller%2E+1998%2E+Efficient+BackProp%2E+In+Neural+Networks%3A+Tricks+of+the+Trade%2C+This+Book+is+an+Outgrowth+of+a+1996+NIPS+Workshop%2C+pages+9%E2%80%9350%2C+)
14. Omer Levy and Yoav Goldberg. 2014. Dependency-Based Word Embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, pages 302–308. [find](https://scholar.google.com/scholar?q=Omer+Levy+and+Yoav+Goldberg%2E+2014%2E+Dependency%2DBased+Word+Embeddings%2E+In+Proceedings+of+the+52nd+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+ACL+2014%2C+June+22%2D27%2C+2014%2C+Baltimore%2C+)
15. Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics (AIS-TATS10). Society for Artificial Intelligence and Statistics. [find](https://scholar.google.com/scholar?q=Xavier+Glorot+and+Yoshua+Bengio%2E+2010%2E+Understanding+the+difficulty+of+training+deep+feedforward+neural+networks%2E+In+In+Proceedings+of+the+International+Conference+on+Artificial+Intelligence+and+Stati)
16. Geoffrey Hinton. 2012. Neural Networks for Machine Learning - Lecture 6a - Overview of mini-batch gradient descent. [find](https://scholar.google.com/scholar?q=Geoffrey+Hinton%2E+2012%2E+Neural+Networks+for+Machine+Learning+%2D+Lecture+6a+%2D+Overview+of+mini%2Dbatch+gradient+descent%2E)
17. Sepp Hochreiter and Jürgen Schmidhuber. 1997a. Flat Minima. Neural Computation, 9(1):1–42. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997a%2E+Flat+Minima%2E+Neural+Computation%2C+9%281%29%3A1%E2%80%9342%2E)
18. Sepp Hochreiter and Jürgen Schmidhuber. 1997b. Long Short-Term Memory. Neural Computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997b%2E+Long+Short%2DTerm+Memory%2E+Neural+Computation%2C+9%288%29%3A1735%E2%80%931780%2E)
19. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF Models for Sequence Tagging. CoRR, abs/1508.01991. [find](https://scholar.google.com/scholar?q=Zhiheng+Huang%2C+Wei+Xu%2C+and+Kai+Yu%2E+2015%2E+Bidirectional+LSTM%2DCRF+Models+for+Sequence+Tagging%2E+CoRR%2C+abs%2F1508%2E01991%2E) [our card](https://md.research.elamur.ai/free/en/papers/huang-2015-bidirectional-lstm-crf-models-sequence/)
20. Adam: CoRR, Qi Li, Heng Ji, and Liang Huang. 2013. Joint Event Extraction via Structured Prediction with Global Features. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 73–82, Sofia, Bulgaria. Association for Computational Li… [find](https://scholar.google.com/scholar?q=Adam%3A+CoRR%2C+Qi+Li%2C+Heng+Ji%2C+and+Liang+Huang%2E+2013%2E+Joint+Event+Extraction+via+Structured+Prediction+with+Global+Features%2E+In+Proceedings+of+the+51st+Annual+Meeting+of+the+Association+for+Computational) [our card](https://md.research.elamur.ai/free/en/papers/li-2013-joint-event-extraction-structured-prediction/)
21. Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF. CoRR, abs/1603.01354. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma+and+Eduard+H%2E+Hovy%2E+2016%2E+End%2Dto%2Dend+Sequence+Labeling+via+Bi%2Ddirectional+LSTM%2DCNNs%2DCRF%2E+CoRR%2C+abs%2F1603%2E01354%2E) [our card](https://md.research.elamur.ai/free/en/papers/ma-2016-end-end-sequence-labeling-bi/)
22. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Comput. Linguist., 19(2):313–330. [find](https://scholar.google.com/scholar?q=Mitchell+P%2E+Marcus%2C+Mary+Ann+Marcinkiewicz%2C+and+Beatrice+Santorini%2E+1993%2E+Building+a+Large+Annotated+Corpus+of+English%3A+The+Penn+Treebank%2E+Comput%2E+Linguist%2E%2C+19%282%29%3A313%E2%80%93330%2E) [our card](https://md.research.elamur.ai/free/en/papers/bank-1993-marcus-building-large-annotated-corpus-english/)
23. Frank J. Massey. 1951. The kolmogorov-smirnov test for goodness of fit. Journal of the American Statistical Association, 46(253):68–78. [find](https://scholar.google.com/scholar?q=Frank+J%2E+Massey%2E+1951%2E+The+kolmogorov%2Dsmirnov+test+for+goodness+of+fit%2E+Journal+of+the+American+Statistical+Association%2C+46%28253%29%3A68%E2%80%9378%2E)
24. Frank Hutter, Holger Hoos, and Kevin Leyton-Brown. 2014. An Efficient Approach for Assessing Hyperparameter Importance. In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, pages I– 754–I–762. JMLR.org. [find](https://scholar.google.com/scholar?q=Frank+Hutter%2C+Holger+Hoos%2C+and+Kevin+Leyton%2DBrown%2E+2014%2E+An+Efficient+Approach+for+Assessing+Hyperparameter+Importance%2E+In+Proceedings+of+the+31st+International+Conference+on+International+Conference+)
25. Tomáš Mikolov. 2012. Statistical language models based on neural networks. Ph.D. thesis, Brno University of Technology. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Mikolov%2E+2012%2E+Statistical+language+models+based+on+neural+networks%2E+Ph%2ED%2E+thesis%2C+Brno+University+of+Technology%2E)
26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Estimation of Word Representations in Vector Space. CoRR, abs/1301.3781. [find](https://scholar.google.com/scholar?q=Tomas+Mikolov%2C+Kai+Chen%2C+Greg+Corrado%2C+and+Jeffrey+Dean%2E+2013%2E+Efficient+Estimation+of+Word+Representations+in+Vector+Space%2E+CoRR%2C+abs%2F1301%2E3781%2E)
27. Yurii Nesterov. 1983. A method of solving a convex programming problem with convergence rate O(1/sqr(k)). Soviet Mathematics Doklady, 27:372– 376. [find](https://scholar.google.com/scholar?q=Yurii+Nesterov%2E+1983%2E+A+method+of+solving+a+convex+programming+problem+with+convergence+rate+O%281%2Fsqr%28k%29%29%2E+Soviet+Mathematics+Doklady%2C+27%3A372%E2%80%93+376%2E)
28. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the Difficulty of Training Recurrent Neural Networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, ICML’13, pages III–1310–III– 1318. JMLR.org. [find](https://scholar.google.com/scholar?q=Razvan+Pascanu%2C+Tomas+Mikolov%2C+and+Yoshua+Bengio%2E+2013%2E+On+the+Difficulty+of+Training+Recurrent+Neural+Networks%2E+In+Proceedings+of+the+30th+International+Conference+on+International+Conference+on+Mach)
29. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+D%2E+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+representation%2E+In+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1532%E2%80%931543%2E)
30. Nils Reimers and Iryna Gurevych. 2017. Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks. arXiv preprint arXiv:1707.06799. — [arXiv:1707.06799](https://arxiv.org/abs/1707.06799) [find](https://scholar.google.com/scholar?q=Nils+Reimers+and+Iryna+Gurevych%2E+2017%2E+Optimal+Hyperparameters+for+Deep+LSTM%2DNetworks+for+Sequence+Labeling+Tasks%2E+arXiv+preprint+arXiv%3A1707%2E06799%2E+%E2%80%94+%5BarXiv%3A1707%2E06799%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1707%2E0679)
31. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 231–235, Berlin, Germany. Association for Computational Linguis… [find](https://scholar.google.com/scholar?q=Anders+S%C3%B8gaard+and+Yoav+Goldberg%2E+2016%2E+Deep+multi%2Dtask+learning+with+low+level+tasks+supervised+at+lower+layers%2E+In+Proceedings+of+the+54th+Annual+Meeting+of+the+Association+for+Computational+Linguis) [our card](https://md.research.elamur.ai/free/en/papers/sogaard-2016-deep-multi-task-learning-low/)
32. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. J. Mach. Learn. Res., 15(1):1929– 1958. [find](https://scholar.google.com/scholar?q=Nitish+Srivastava%2C+Geoffrey+Hinton%2C+Alex+Krizhevsky%2C+Ilya+Sutskever%2C+and+Ruslan+Salakhutdinov%2E+2014%2E+Dropout%3A+A+Simple+Way+to+Prevent+Neural+Networks+from+Overfitting%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+15%281%29%3A1929)
33. Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich Part-of-speech Tagging with a Cyclic Dependency Network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Vol… [find](https://scholar.google.com/scholar?q=Kristina+Toutanova%2C+Dan+Klein%2C+Christopher+D%2E+Manning%2C+and+Yoram+Singer%2E+2003%2E+Feature%2Drich+Part%2Dof%2Dspeech+Tagging+with+a+Cyclic+Dependency+Network%2E+In+Proceedings+of+the+2003+Conference+of+the+North+) [our card](https://md.research.elamur.ai/free/en/papers/toutanova-2003-feature-rich-part-speech-tagging/)
34. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… [find](https://scholar.google.com/scholar?q=Yonghui+Wu%2C+Mike+Schuster%2C+Zhifeng+Chen%2C+Quoc+V%2E+Le%2C+Mohammad+Norouzi%2C+Wolfgang+Macherey%2C+Maxim+Krikun%2C+Yuan+Cao%2C+Qin+Gao%2C+Klaus+Macherey%2C+Jeff+Klingner%2C+Apurva+Shah%2C+Melvin+Johnson%2C+Xiaobing+Liu%2C+Luk)
35. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701. [find](https://scholar.google.com/scholar?q=Matthew+D%2E+Zeiler%2E+2012%2E+ADADELTA%3A+an+adaptive+learning+rate+method%2E+CoRR%2C+abs%2F1212%2E5701%2E) [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)