--- title: "Neural Machine Translation by Jointly — Bahdanau 2014" description: "The authors work on neural machine translation. In the encoder–decoder models of the time, the entire source sentence was compressed into a single…" authors: ["Dzmitry Bahdanau", "Kyunghyun Cho", "Yoshua Bengio"] year: 2014 venue: "arXiv:1409.0473" source: "https://arxiv.org/abs/1409.0473" doi: "10.48550/arxiv.1409.0473" arxiv: "1409.0473" licence: "arXiv — non-exclusive licence to distribute" licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / Neural Machine Translation by Jointly Learning to Align and Translate # Neural Machine Translation by Jointly Learning to Align and Translate **Authors:** [Dzmitry Bahdanau](https://md.research.elamur.ai/free/en/authors/dzmitry-bahdanau/), [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/), [Yoshua Bengio](https://md.research.elamur.ai/free/en/authors/yoshua-bengio/) · **Year:** 2014 · **Venue:** arXiv:1409.0473 · **Licence:** [arXiv — non-exclusive licence to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/) **Links:** [Publisher's page](https://arxiv.org/abs/1409.0473) · [PDF](https://arxiv.org/pdf/1409.0473) · [DOI 10.48550/arxiv.1409.0473](https://doi.org/10.48550/arxiv.1409.0473) · [arXiv 1409.0473](https://arxiv.org/abs/1409.0473) · [OpenAlex](https://openalex.org/W2133564696) **Citations:** 14130 citations (openalex-snapshot · 2026-10-14) · [4778](#cited-by) in our library, of the 140884 papers whose bibliographies we read ## Summary The authors work on neural machine translation. In the encoder–decoder models of the time, the entire source sentence was compressed into a single fixed-length vector. As a result, quality dropped quickly as sentences grew longer. The authors propose the RNNsearch architecture. In it, a bidirectional RNN encodes each source word as a separate annotation that takes into account both the left and the right context. When generating each word of the translation, the decoder itself chooses which parts of the input sentence to attend to. The weights are computed by a small feedforward alignment network, which is trained jointly with the whole model. This is how the attention mechanism in its modern sense came about. The model was evaluated on English-to-French translation using the ACL WMT '14 parallel corpora. RNNsearch clearly outperforms the baseline RNN Encoder–Decoder, and the gap is especially large on long sentences. The quality of a single model approaches that of the phrase-based statistical translation system Moses. The attention weight matrices show word alignments that look linguistically plausible. This work introduced the attention mechanisms from which the Transformer later grew. ## Key points - Models: RNNenc (the baseline encoder–decoder) and RNNsearch (with attention). Each was trained on sentences of up to 30 and up to 50 words (RNNsearch-30/-50, RNNenc-30/-50). The comparison system is the phrase-based Moses. - Data: the WMT '14 parallel corpora for the English→French pair. The vocabulary was limited to roughly 30,000 most frequent words in each language. - BLEU on the test set (from the omitted part of the excerpt, taken from the known version of the paper): RNNsearch-50 — 26.75 on all sentences and 34.16 on sentences without unknown words. RNNenc-50 scores 17.82 and 26.71, respectively. For Moses on sentences without unknown words — 35.63, and for the longer-trained RNNsearch-50* — 36.15. - Hyperparameters: hidden layer 1000, embeddings 620, maxout layer 500, 1000 hidden units in the alignment model. Optimization: SGD with Adadelta, minibatches of 80 sentences, gradients clipped at norm 1. - On sentences of 30 words or more, RNNsearch-50 translates coherently and accurately, while RNNenc-50 loses or distorts the meaning. For the authors, soft alignment as a differentiable alternative to hard latent alignment made it possible to train the system as a single whole. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Linguistics & languages](https://md.research.elamur.ai/free/en/shelves/linguistics/) · [Machine translation](https://md.research.elamur.ai/free/en/shelves/machine-translation/) · [AI & machine learning](https://md.research.elamur.ai/free/en/shelves/ai/) Topics: [Machine translation](https://md.research.elamur.ai/free/en/topics/machine-translation/) ## Cited by Cited by 4778 papers of our library cite it (of the 140884 whose bibliographies we read); 437 of them have a page here, the first 100 below, 4341 not yet. - [BPoMP: The Benchmark of Poetic Minimal Pairs – Limericks, Rhyme, and Narrative Coherence](https://md.research.elamur.ai/free/en/papers/abdibayev-2021-bpomp-benchmark-poetic-minimal-pairs/) - [Quantifying Attention Flow in Transformers](https://md.research.elamur.ai/free/en/papers/abnar-2020-quantifying-attention-flow-transformers/) - [Historical Text Normalization with Delayed Rewards](https://md.research.elamur.ai/free/en/papers/acl-2019-flachs-historical-text-normalization-delayed-rewards/) - [Joint Diacritization, Lemmatization, Normalization, and Fine-Grained Morphological Tagging](https://md.research.elamur.ai/free/en/papers/acl-2020-zalmout-joint-diacritization-lemmatization-normalization-fine/) - [Morphological Word Embeddings for Arabic Neural Machine Translation in Low-Resource Settings](https://md.research.elamur.ai/free/en/papers/aclbulk-2018-shapiro-morphological-word-embeddings-arabic-neural/) - [Few-Shot and Zero-Shot Learning for Historical Text Normalization](https://md.research.elamur.ai/free/en/papers/aclbulk-2019-bollmann-few-shot-zero-shot-learning/) - [Neural Arabic Text Diacritization: State of the Art Results and a Novel Approach for Machine Translation](https://md.research.elamur.ai/free/en/papers/aclbulk-2019-fadel-neural-arabic-text-diacritization-state/) - [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://md.research.elamur.ai/free/en/papers/adi-2016-fine-grained-analysis-sentence-embeddings/) - [Towards a Human-like Open-Domain Chatbot](https://md.research.elamur.ai/free/en/papers/adiwardana-2020-towards-human-like-open-domain/) - [Semantically Corrected Amharic Automatic Speech Recognition](https://md.research.elamur.ai/free/en/papers/adnew-2024-semantically-corrected-amharic-automatic-speech/) - [Morphological Inflection Generation with Hard Monotonic Attention](https://md.research.elamur.ai/free/en/papers/aharoni-2017-morphological-inflection-generation-hard-monotonic/) - [Towards String-To-Tree Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2017-string-tree-neural-machine-translation/) - [Massively Multilingual Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/aharoni-2019-massively-multilingual-neural-machine-translation/) - [Domain Adaptation for Arabic Machine Translation: The Case of Financial Texts](https://md.research.elamur.ai/free/en/papers/alghamdi-2023-domain-adaptation-arabic-machine-translation/) - [The Falcon Series of Open Language Models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/) - [ANETAC: Arabic Named Entity Transliteration and Classification Dataset](https://md.research.elamur.ai/free/en/papers/ameur-2019-anetac-arabic-named-entity-transliteration/) - [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://md.research.elamur.ai/free/en/papers/amodei-2015-deep-speech-2-end-end/) - [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://md.research.elamur.ai/free/en/papers/anderson-2017-vision-language-navigation-interpreting-visually/) - [Good-Enough Compositional Data Augmentation](https://md.research.elamur.ai/free/en/papers/andreas-2020-good-enough-compositional-data-augmentation/) - [Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-massively-multilingual-neural-machine-translation/) - [Monotonic Infinite Lookback Attention for Simultaneous Machine Translation](https://md.research.elamur.ai/free/en/papers/arivazhagan-2019-monotonic-infinite-lookback-attention-simultaneous/) - [Unsupervised Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/artetxe-2017-unsupervised-neural-machine-translation/) - [Incorporating Discrete Translation Lexicons into Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/arthur-2016-incorporating-discrete-translation-lexicons-into/) - [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://md.research.elamur.ai/free/en/papers/augenstein-2019-multifc-real-world-multi-domain/) - [Take the Hint: Improving Arabic Diacritization with Partially-Diacritized Text](https://md.research.elamur.ai/free/en/papers/bahar-2023-take-hint-improving-arabic-diacritization/) - [An Actor-Critic Algorithm for Sequence Prediction](https://md.research.elamur.ai/free/en/papers/bahdanau-2016-actor-critic-algorithm-sequence-prediction/) - [Graph Pre-training for AMR Parsing and Generation](https://md.research.elamur.ai/free/en/papers/bai-2022-graph-pre-training-amr-parsing/) - [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://md.research.elamur.ai/free/en/papers/bajaj-2016-ms-marco-human-generated/) - [Simple, Scalable Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bapna-2019-simple-scalable-adaptation-neural-machine/) - [Findings of the Third Shared Task on Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/barrault-2018-findings-third-shared-task-multimodal/) - [Findings of the 2019 Conference on Machine Translation (WMT19)](https://md.research.elamur.ai/free/en/papers/barrault-2019-findings-2019-conference-machine-translation/) - [Graph Convolutional Encoders for Syntax-aware Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bastings-2017-graph-convolutional-encoders-syntax-aware/) - [Relational inductive biases, deep learning, and graph networks](https://md.research.elamur.ai/free/en/papers/battaglia-2018-relational-inductive-biases-deep-learning/) - [Evaluating Discourse Phenomena in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/bawden-2017-evaluating-discourse-phenomena-neural-machine/) - [Automatic Evaluation and Analysis of Idioms in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/baziotis-2023-automatic-evaluation-analysis-idioms-neural/) - [What do Neural Machine Translation Models Learn about Morphology?](https://md.research.elamur.ai/free/en/papers/belinkov-2017-what-do-neural-machine-translation/) - [Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks](https://md.research.elamur.ai/free/en/papers/belinkov-2018-evaluating-layers-representation-neural-machine/) - [Analysis Methods in Neural Language Processing: A Survey](https://md.research.elamur.ai/free/en/papers/belinkov-2019-analysis-methods-neural-language-processing/) - [Neural versus Phrase-Based Machine Translation Quality: a Case Study](https://md.research.elamur.ai/free/en/papers/bentivogli-2016-neural-versus-phrase-based-machine/) - [Findings of the 2017 Conference on Machine Translation (WMT17)](https://md.research.elamur.ai/free/en/papers/bojar-2017-findings-2017-conference-machine-translation/) - [Results of the WMT17 Metrics Shared Task](https://md.research.elamur.ai/free/en/papers/bojar-2017-results-wmt17-metrics-shared-task/) - [Findings of the 2018 Conference on Machine Translation (WMT18)](https://md.research.elamur.ai/free/en/papers/bojar-2018-findings-2018-conference-machine-translation/) - [On the Opportunities and Risks of Foundation Models](https://md.research.elamur.ai/free/en/papers/bommasani-2021-opportunities-risks-foundation-models/) - [Generating Sentences from a Continuous Space](https://md.research.elamur.ai/free/en/papers/bowman-2016-generating-sentences-continuous-space/) - [Effective Domain Mixing for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/britz-2017-effective-domain-mixing-neural-machine/) - [Massive Exploration of Neural Machine Translation Architectures](https://md.research.elamur.ai/free/en/papers/britz-2017-massive-exploration-neural-machine-translation/) - [Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset](https://md.research.elamur.ai/free/en/papers/byrne-2019-taskmaster-1-realistic-diverse-dialog/) - [Probing the Need for Visual Context in Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/caglayan-2019-probing-need-visual-context-multimodal/) - [e-SNLI: Natural Language Inference with Natural Language Explanations](https://md.research.elamur.ai/free/en/papers/camburu-2018-e-snli-natural-language-inference/) - [Factorising AMR generation through syntax](https://md.research.elamur.ai/free/en/papers/cao-2019-factorising-amr-generation-through-syntax/) - [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://md.research.elamur.ai/free/en/papers/carlini-2018-secret-sharer-evaluating-testing-unintended/) - [Extracting Training Data from Large Language Models](https://md.research.elamur.ai/free/en/papers/carlini-2020-extracting-training-data-large-language/) - [Evaluating prose style transfer with the Bible](https://md.research.elamur.ai/free/en/papers/carlson-2017-evaluating-prose-style-transfer-bible/) - [Emerging Properties in Self-Supervised Vision Transformers](https://md.research.elamur.ai/free/en/papers/caron-2021-emerging-properties-self-supervised-vision/) - [Tagged Back-Translation](https://md.research.elamur.ai/free/en/papers/caswell-2019-tagged-back-translation/) - [Deep Communicating Agents for Abstractive Summarization](https://md.research.elamur.ai/free/en/papers/celikyilmaz-2018-deep-communicating-agents-abstractive-summarization/) - [HATFormer: Historic Handwritten Arabic Text Recognition with Transformers](https://md.research.elamur.ai/free/en/papers/chan-2024-hatformer-historic-handwritten-arabic-text/) - [Findings of the WMT 2018 Shared Task on Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/chatterjee-2018-findings-wmt-2018-shared-task/) - [Findings of the WMT 2019 Shared Task on Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/chatterjee-2019-findings-wmt-2019-shared-task/) - [Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder](https://md.research.elamur.ai/free/en/papers/chen-2017-improved-neural-machine-translation-syntax/) - [Reading Wikipedia to Answer Open-Domain Questions](https://md.research.elamur.ai/free/en/papers/chen-2017-reading-wikipedia-answer-open-domain/) - [Recurrent Attention Network on Memory for Aspect Sentiment Analysis](https://md.research.elamur.ai/free/en/papers/chen-2017-recurrent-attention-network-memory-aspect/) - [The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chen-2018-best-both-worlds-combining-recent/) - [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/) - [Iterative Translation Refinement with Large Language Models](https://md.research.elamur.ai/free/en/papers/chen-2024-iterative-translation-refinement-large-language/) - [Long Short-Term Memory-Networks for Machine Reading](https://md.research.elamur.ai/free/en/papers/cheng-2016-long-short-term-memory-networks/) - [Neural Summarization by Extracting Sentences and Words](https://md.research.elamur.ai/free/en/papers/cheng-2016-neural-summarization-extracting-sentences-words/) - [Robust Neural Machine Translation with Doubly Adversarial Inputs](https://md.research.elamur.ai/free/en/papers/cheng-2019-robust-neural-machine-translation-doubly/) - [Abstractive Sentence Summarization with Attentive Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/chopra-2016-abstractive-sentence-summarization-attentive-recurrent/) - [Attention-Based Models for Speech Recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/) - [A Survey of Domain Adaptation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chu-2018-survey-domain-adaptation-neural-machine/) - [A Character-level Decoder without Explicit Segmentation for Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/chung-2016-character-level-decoder-without-explicit/) - [Let's do it "again": A First Computational Approach to Detecting Adverbial Presupposition Triggers](https://md.research.elamur.ai/free/en/papers/cianflone-2018-let-s-do-it-again/) - [Semi-Supervised Sequence Modeling with Cross-View Training](https://md.research.elamur.ai/free/en/papers/clark-2018-semi-supervised-sequence-modeling-cross/) - [A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents](https://md.research.elamur.ai/free/en/papers/cohan-2018-discourse-aware-attention-model-abstractive/) - [Are we there yet? Encoder-decoder neural networks as cognitive models of English past tense inflection](https://md.research.elamur.ai/free/en/papers/corkery-2019-are-we-there-yet-encoder/) - [Adaptively Sparse Transformers](https://md.research.elamur.ai/free/en/papers/correia-2019-adaptively-sparse-transformers/) - [Copied Monolingual Data Improves Low-Resource Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/currey-2017-copied-monolingual-data-improves-low/) - [A Regularization Approach for Incorporating Event Knowledge and Coreference Relations into Neural Discourse Parsing](https://md.research.elamur.ai/free/en/papers/dai-2019-regularization-approach-incorporating-event-knowledge/) - [Style Transformer: Unpaired Text Style Transfer without Disentangled Latent Representation](https://md.research.elamur.ai/free/en/papers/dai-2019-style-transformer-unpaired-text-style/) - [A Survey of the State of Explainable AI for Natural Language Processing](https://md.research.elamur.ai/free/en/papers/danilevsky-2020-survey-state-explainable-ai-natural/) - [Extractive Summarization for Arabic Documents Using SAraBERT with a Semantic Siamese Similarity Evaluation Metric](https://md.research.elamur.ai/free/en/papers/deen-2026-extractive-summarization-arabic-documents-using/) - [Stronger Baselines for Trustable Results in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/denkowski-2017-stronger-baselines-trustable-results-neural/) - [A Distributional and Orthographic Aggregation Model for English Derivational Morphology](https://md.research.elamur.ai/free/en/papers/deutsch-2018-distributional-orthographic-aggregation-model-english/) - [ERASER: A Benchmark to Evaluate Rationalized NLP Models](https://md.research.elamur.ai/free/en/papers/deyoung-2020-eraser-benchmark-evaluate-rationalized-nlp/) - [Word-Level Coreference Resolution](https://md.research.elamur.ai/free/en/papers/dobrovolskii-2021-word-level-coreference-resolution/) - [Improving Text-to-SQL Evaluation Methodology](https://md.research.elamur.ai/free/en/papers/dollak-2018-improving-text-sql-evaluation-methodology/) - [Spelling Normalization of Historical Documents by Using a Machine Translation Approach](https://md.research.elamur.ai/free/en/papers/domingo-2018-spelling-normalization-historical-documents-machine/) - [Multi-Task Learning for Multiple Language Translation](https://md.research.elamur.ai/free/en/papers/dong-2015-multi-task-learning-multiple-language/) - [Learning to Paraphrase for Question Answering](https://md.research.elamur.ai/free/en/papers/dong-2017-learning-paraphrase-question-answering/) - [Is Neural Machine Translation Ready for Deployment? A Case Study on 30 Translation Directions](https://md.research.elamur.ai/free/en/papers/dowmunt-2016-is-neural-machine-translation-ready/) - [Log-linear Combinations of Monolingual and Bilingual Neural Machine Translation Models for Automatic Post-Editing](https://md.research.elamur.ai/free/en/papers/dowmunt-2016-log-linear-combinations-monolingual-bilingual/) - [Learning to Ask: Neural Question Generation for Reading Comprehension](https://md.research.elamur.ai/free/en/papers/du-2017-learning-ask-neural-question-generation/) - [Question Generation for Question Answering](https://md.research.elamur.ai/free/en/papers/duan-2017-question-generation-question-answering/) - [SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine](https://md.research.elamur.ai/free/en/papers/dunn-2017-searchqa-new-q-dataset-augmented/) - [Semantic Noise Matters for Neural Natural Language Generation](https://md.research.elamur.ai/free/en/papers/dusek-2019-semantic-noise-matters-neural-natural/) - [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://md.research.elamur.ai/free/en/papers/edunov-2018-classical-structured-prediction-losses-sequence/) - [On The Evaluation of Machine Translation Systems Trained With Back-Translation](https://md.research.elamur.ai/free/en/papers/edunov-2020-evaluation-machine-translation-systems-trained/) - [Neural End-to-End Learning for Computational Argumentation Mining](https://md.research.elamur.ai/free/en/papers/eger-2017-neural-end-end-learning-computational/) - [Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation](https://md.research.elamur.ai/free/en/papers/eikema-2020-is-map-decoding-all-you/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @misc{bahdanau2014neural, title = {{Neural Machine Translation by Jointly Learning to Align and Translate}}, author = {Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua}, year = {2014}, doi = {10.48550/arxiv.1409.0473}, eprint = {1409.0473}, archivePrefix = {arXiv}, url = {https://doi.org/10.48550/arxiv.1409.0473} } ``` - [← previous Negation scope and spelling variation for text-mining of Danish electronic patient records](https://md.research.elamur.ai/free/en/papers/aclbulk-2014-thomas-negation-scope-spelling-variation-text/) - [next → New Technologies for Old Germanic. Resources and Research on Parallel Bibles in Older Continental Western Germanic](https://md.research.elamur.ai/free/en/papers/chiarcos-2014-new-technologies-old-germanic-resources/) - [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) ## References 30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Axelrod, A., He, X., and Gao, J. (2011). Domain adaptation via pseudo in-domain data selection. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 355–362. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Axelrod%2C+A%2E%2C+He%2C+X%2E%2C+and+Gao%2C+J%2E+%282011%29%2E+Domain+adaptation+via+pseudo+in%2Ddomain+data+selection%2E+In+Proceedings+of+the+ACL+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+) [our card](https://md.research.elamur.ai/free/en/papers/axelrod-2011-domain-adaptation-pseudo-domain-data/) 2. Bastien, F., Lamblin, P., Pascanu, R., Bergstra, J., Goodfellow, I. J., Bergeron, A., Bouchard, N., and Bengio, Y. (2012). Theano: new features and speed improvements. [↗](https://doi.org/10.48550/arxiv.1211.5590) 3. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. [find](https://scholar.google.com/scholar?q=Deep+Learning+and+Unsupervised+Feature+Learning+NIPS+2012+Workshop%2E) 4. Bengio, Y., Simard, P., and Frasconi, P. (1994). Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2), 157–166. [↗](https://doi.org/10.1109/72.279181) 5. Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model. J. Mach. Learn. Res., 3, 1137–1155. [find](https://scholar.google.com/scholar?q=Bengio%2C+Y%2E%2C+Ducharme%2C+R%2E%2C+Vincent%2C+P%2E%2C+and+Janvin%2C+C%2E+%282003%29%2E+A+neural+probabilistic+language+model%2E+J%2E+Mach%2E+Learn%2E+Res%2E%2C+3%2C+1137%E2%80%931155%2E) 6. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde-Farley, D., and Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy). Oral Presentation. [find](https://scholar.google.com/scholar?q=Bergstra%2C+J%2E%2C+Breuleux%2C+O%2E%2C+Bastien%2C+F%2E%2C+Lamblin%2C+P%2E%2C+Pascanu%2C+R%2E%2C+Desjardins%2C+G%2E%2C+Turian%2C+J%2E%2C+Warde%2DFarley%2C+D%2E%2C+and+Bengio%2C+Y%2E+%282010%29%2E+Theano%3A+a+CPU+and+GPU+math+expression+compiler%2E+In+Proceedings+o) 7. Boulanger-Lewandowski, N., Bengio, Y., and Vincent, P. (2013). Audio chord recognition with recurrent neural networks. In ISMIR. [find](https://scholar.google.com/scholar?q=Boulanger%2DLewandowski%2C+N%2E%2C+Bengio%2C+Y%2E%2C+and+Vincent%2C+P%2E+%282013%29%2E+Audio+chord+recognition+with+recurrent+neural+networks%2E+In+ISMIR%2E) 8. Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., and Bengio, Y. (2014a). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014). to appear. [↗](https://doi.org/10.48550/arxiv.1406.1078) 9. Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014b). On the properties of neural machine translation: Encoder–Decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. to appear. [find](https://scholar.google.com/scholar?q=Cho%2C+K%2E%2C+van+Merrie%CC%88nboer%2C+B%2E%2C+Bahdanau%2C+D%2E%2C+and+Bengio%2C+Y%2E+%282014b%29%2E+On+the+properties+of+neural+machine+translation%3A+Encoder%E2%80%93Decoder+approaches%2E+In+Eighth+Workshop+on+Syntax%2C+Semantics+and+Structure+) [our card](https://md.research.elamur.ai/free/en/papers/cho-2014-properties-neural-machine-translation-encoder/) 10. Devlin, J., Zbib, R., Huang, Z., Lamar, T., Schwartz, R., and Makhoul, J. (2014). Fast and robust neural network joint models for statistical machine translation. In Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Devlin%2C+J%2E%2C+Zbib%2C+R%2E%2C+Huang%2C+Z%2E%2C+Lamar%2C+T%2E%2C+Schwartz%2C+R%2E%2C+and+Makhoul%2C+J%2E+%282014%29%2E+Fast+and+robust+neural+network+joint+models+for+statistical+machine+translation%2E+In+Association+for+Computational+Ling) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/) 11. Forcada, M. L. and Ñeco, R. P. (1997). Recursive hetero-associative memories for translation. In J. Mira, R. Moreno-Dı́az, and J. Cabestany, editors, Biological and Artificial Computation: From Neuroscience to Technology, volume 1240 of Lecture Notes in Computer Science, pages 453–462. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Forcada%2C+M%2E+L%2E+and+N%CC%83eco%2C+R%2E+P%2E+%281997%29%2E+Recursive+hetero%2Dassociative+memories+for+translation%2E+In+J%2E+Mira%2C+R%2E+Moreno%2DD%C4%B1%CC%81az%2C+and+J%2E+Cabestany%2C+editors%2C+Biological+and+Artificial+Computation%3A+From+Neuro) 12. Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. (2013). Maxout networks. In Proceedings of The 30th International Conference on Machine Learning, pages 1319– 1327. [find](https://scholar.google.com/scholar?q=Goodfellow%2C+I%2E%2C+Warde%2DFarley%2C+D%2E%2C+Mirza%2C+M%2E%2C+Courville%2C+A%2E%2C+and+Bengio%2C+Y%2E+%282013%29%2E+Maxout+networks%2E+In+Proceedings+of+The+30th+International+Conference+on+Machine+Learning%2C+pages+1319%E2%80%93+1327%2E) 13. Graves, A. (2012). Sequence transduction with recurrent neural networks. In Proceedings of the 29th International Conference on Machine Learning (ICML 2012). [↗](https://doi.org/10.48550/arxiv.1211.3711) [our card](https://md.research.elamur.ai/free/en/papers/graves-2012-sequence-transduction-recurrent-neural-networks/) 14. Graves, A. (2013). Generating sequences with recurrent neural networks. arXiv:1308.0850 [cs.NE]. [↗](https://arxiv.org/abs/1308.0850) 15. Graves, A., Jaitly, N., and Mohamed, A.-R. (2013). Hybrid speech recognition with deep bidirectional LSTM. In Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, pages 273–278. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E%2C+Jaitly%2C+N%2E%2C+and+Mohamed%2C+A%2E%2DR%2E+%282013%29%2E+Hybrid+speech+recognition+with+deep+bidirectional+LSTM%2E+In+Automatic+Speech+Recognition+and+Understanding+%28ASRU%29%2C+2013+IEEE+Workshop+on%2C+pages+273%E2%80%9327) 16. Hermann, K. and Blunsom, P. (2014). Multilingual distributed representations without word alignment. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014). [↗](https://doi.org/10.48550/arxiv.1312.6173) 17. Hochreiter, S. (1991). Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut für Informatik, Lehrstuhl Prof. Brauer, Technische Universität München. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+%281991%29%2E+Untersuchungen+zu+dynamischen+neuronalen+Netzen%2E+Diploma+thesis%2C+Institut+fu%CC%88r+Informatik%2C+Lehrstuhl+Prof%2E+Brauer%2C+Technische+Universita%CC%88t+Mu%CC%88nchen%2E) 18. Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+%281997%29%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%2C+1735%E2%80%931780%2E) 19. Kalchbrenner, N. and Blunsom, P. (2013). Recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700–1709. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kalchbrenner%2C+N%2E+and+Blunsom%2C+P%2E+%282013%29%2E+Recurrent+continuous+translation+models%2E+In+Proceedings+of+the+ACL+Conference+on+Empirical+Methods+in+Natural+Language+Processing+%28EMNLP%29%2C+pages+1700%E2%80%931709%2E+Ass) [our card](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/) 20. Koehn, P. (2010). Statistical Machine Translation. Cambridge University Press, New York, NY, USA. [↗](https://doi.org/10.1017/cbo9780511815829) 21. Koehn, P., Och, F. J., and Marcu, D. (2003). Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Koehn%2C+P%2E%2C+Och%2C+F%2E+J%2E%2C+and+Marcu%2C+D%2E+%282003%29%2E+Statistical+phrase%2Dbased+translation%2E+In+Proceedings+of+the+2003+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+) 22. Pascanu, R., Mikolov, T., and Bengio, Y. (2013a). On the difficulty of training recurrent neural networks. In ICML’2013. [↗](https://doi.org/10.48550/arxiv.1211.5063) 23. Pascanu, R., Mikolov, T., and Bengio, Y. (2013b). On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013). [↗](https://doi.org/10.48550/arxiv.1211.5063) 24. Pascanu, R., Gulcehre, C., Cho, K., and Bengio, Y. (2014). How to construct deep recurrent neural networks. In Proceedings of the Second International Conference on Learning Representations (ICLR 2014). [↗](https://doi.org/10.48550/arxiv.1312.6026) 25. Pouget-Abadie, J., Bahdanau, D., van Merriënboer, B., Cho, K., and Bengio, Y. (2014). Overcoming the curse of sentence length for neural machine translation using automatic segmentation. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. to appear. [find](https://scholar.google.com/scholar?q=Pouget%2DAbadie%2C+J%2E%2C+Bahdanau%2C+D%2E%2C+van+Merrie%CC%88nboer%2C+B%2E%2C+Cho%2C+K%2E%2C+and+Bengio%2C+Y%2E+%282014%29%2E+Overcoming+the+curse+of+sentence+length+for+neural+machine+translation+using+automatic+segmentation%2E+In+Eighth+Wo) 26. Schuster, M. and Paliwal, K. K. (1997). Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11), 2673–2681. [↗](https://doi.org/10.1109/78.650093) 27. Schwenk, H. (2012). Continuous space translation models for phrase-based statistical machine translation. In M. Kay and C. Boitet, editors, Proceedings of the 24th International Conference on Computational Linguistics (COLIN), pages 1071–1080. Indian Institute of Technology Bombay. [find](https://scholar.google.com/scholar?q=Schwenk%2C+H%2E+%282012%29%2E+Continuous+space+translation+models+for+phrase%2Dbased+statistical+machine+translation%2E+In+M%2E+Kay+and+C%2E+Boitet%2C+editors%2C+Proceedings+of+the+24th+International+Conference+on+Computat) 28. Schwenk, H., Dchelotte, D., and Gauvain, J.-L. (2006). Continuous space language models for statistical machine translation. In Proceedings of the COLING/ACL on Main conference poster sessions, pages 723–730. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Schwenk%2C+H%2E%2C+Dchelotte%2C+D%2E%2C+and+Gauvain%2C+J%2E%2DL%2E+%282006%29%2E+Continuous+space+language+models+for+statistical+machine+translation%2E+In+Proceedings+of+the+COLING%2FACL+on+Main+conference+poster+sessions%2C+pages+) 29. Sutskever, I., Vinyals, O., and Le, Q. (2014). Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems (NIPS 2014). [↗](https://doi.org/10.48550/arxiv.1409.3215) 30. ADADELTA: An adaptive learning rate method. arXiv:1212.5701 [↗](https://arxiv.org/abs/1212.5701) [our card](https://md.research.elamur.ai/free/en/papers/zeiler-2012-adadelta-adaptive-learning-rate-method/)