---
title: "Insertion-based Decoding with Automatically Inferred Generation"
description: "Gu, Liu and Cho propose InDIGO, a decoding scheme that treats the generation order as a hidden… 55 references, cited by 41 papers here."
authors: ["Jiatao Gu", "Qi Liu", "Kyunghyun Cho"]
year: 2019
venue: "Transactions of the Association for Computational Linguistics"
source: "https://aclanthology.org/Q19-1042/"
doi: "10.1162/tacl_a_00292"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/gu-2019-insertion-based-decoding-automatically-inferred/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/gu-2019-insertion-based-decoding-automatically-inferred/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/) / Insertion-based Decoding with Automatically Inferred Generation Order

# Insertion-based Decoding with Automatically Inferred Generation Order

**Authors:** [Jiatao Gu](https://md.research.elamur.ai/free/en/authors/jiatao-gu/), [Qi Liu](https://md.research.elamur.ai/free/en/authors/qi-liu/) and [Kyunghyun Cho](https://md.research.elamur.ai/free/en/authors/kyunghyun-cho/) · **Year:** 2019 · **Venue:** [Transactions of the Association for Computational Linguistics](https://md.research.elamur.ai/free/en/venues/tacl/) ([Transactions of the Association for Computational Linguistics (TACL) 2019](https://md.research.elamur.ai/free/en/venues/tacl/2019/)) · **Publisher:** [The MIT Press](https://md.research.elamur.ai/free/en/publishers/the-mit-press/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/Q19-1042/) · [PDF](https://aclanthology.org/Q19-1042.pdf) · [DOI 10.1162/tacl\_a\_00292](https://doi.org/10.1162/tacl_a_00292) · [OpenAlex](https://openalex.org/W2985694911)

**Citations:** 84 citations (OpenAlex, 2026-10-14) · [41](https://md.research.elamur.ai/free/en/papers/gu-2019-insertion-based-decoding-automatically-inferred/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Standard autoregressive decoders write output strictly from left to right, a convention that may not suit every task. Gu, Liu and Cho propose InDIGO, a decoding scheme that treats the generation order as a hidden variable. At every step, the model chooses the next token and also where it goes, placing it to the left or right of a word already produced. Positions are stored as relative relations (left, same, right) rather than absolute indices. Because these relations stay fixed once created, earlier hidden states can be reused without recomputation. The authors build this into a modified Transformer decoder, called Transformer-InDIGO, and train it by maximizing an ELBO. They compare two approximate posteriors over orders: fixed predefined orders and adaptive orders found by beam search, the latter called SAO. Experiments cover word order recovery, machine translation, image captioning and code generation. Across these tasks, the arbitrary-order decoder matches or beats left-to-right baselines, and the orders it learns depend on the input. This suggests that left-to-right generation is a convenient default rather than a requirement.

## Key points
- Each token's position is encoded as a ternary vector with values in {−1, 0, 1} relative to all other tokens, and these vectors map back to absolute positions at the end. In self-attention, this works as a relative-position bias in the style of Shaw et al. (2018), restricted to distance d = 1.
- Decoding begins with the two boundary tokens ⟨s⟩ and ⟨/s⟩ and ends when a special end-of-decoding token is predicted. The model predicts the word first and then its insertion slot, which worked slightly better than the reverse ordering in preliminary tests.
- Evaluation covers four tasks, including translation examples from an En-Ja dataset. With SAO, translation orders stay fairly close to left-to-right but rearrange whole chunks. Captions and code are mostly produced left to right, with a few words inserted later.
- Compared with the concurrent work of Stern et al. (2019), InDIGO avoids re-encoding the partial sequence at each step and reaches left-to-right quality without knowledge distillation. Unlike Welleck et al. (2019), its learned orders outperform left-to-right on machine translation.
- The code was announced for release in the Fairseq repository. Proposed future work includes a trainable permutation-inference model and adding deletion and substitution operations for post-editing tasks.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 183 works · 228,558 citations
- [New York University](https://md.research.elamur.ai/free/en/universities/new-york-university/) — University, New York, United States · 201 works · 100,180 citations
- [University of Oxford](https://md.research.elamur.ai/free/en/universities/university-of-oxford/) — University, Oxford, United Kingdom · 104 works · 157,183 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

## Cite

Cite as: Jiatao Gu, Qi Liu and Kyunghyun Cho. 2019. Insertion-based Decoding with Automatically Inferred Generation Order. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@article{gu2019insertion,
  title = {{Insertion-based Decoding with Automatically Inferred Generation Order}},
  author = {Jiatao Gu, Qi Liu and Kyunghyun Cho},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.1162/tacl\_a\_00292},
  url = {https://doi.org/10.1162/tacl_a_00292}
}
```

- [← previous Grammar Error Correction in Morphologically Rich Languages: The Case of Russian](https://md.research.elamur.ai/free/en/papers/rozovskaya-2019-grammar-error-correction-morphologically-rich/)
- [next → Broad-Coverage Semantic Parsing as Transduction](https://md.research.elamur.ai/free/en/papers/zhang-2019-broad-coverage-semantic-parsing-as/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/5/)

## References

55 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Roee Aharoni and Yoav Goldberg. 2017. [Towards string-to-tree neural machine translation](https://md.research.elamur.ai/free/en/papers/aharoni-2017-string-tree-neural-machine-translation/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 132–140.
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
3. Satanjeev Banerjee and Alon Lavie. 2005. [Meteor: An automatic metric for MT evaluation with improved correlation with human judgments](https://md.research.elamur.ai/free/en/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/). In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72.
4. Eugene Charniak, Kevin Knight, and Kenji Yamada. 2003. Syntax-based language models for statistical machine translation. In MT Summit IX. Intl. Assoc. for Machine Translation. Citeseer. [find](https://scholar.google.com/scholar?q=Eugene+Charniak%2C+Kevin+Knight%2C+and+Kenji+Yamada%2E+2003%2E+Syntax%2Dbased+language+models+for+statistical+machine+translation%2E+In+MT+Summit+IX%2E+Intl%2E+Assoc%2E+for+Machine+Translation%2E+Citeseer%2E)
5. David Chiang. 2005. [A hierarchical phrase-based model for statistical machine translation](https://md.research.elamur.ai/free/en/papers/chiang-2005-hierarchical-phrase-based-model-statistical/). In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 263–270. Association for Computational Linguistics.
6. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. 2016. [Incorporating copying mechanism in sequence-to-sequence learning](https://md.research.elamur.ai/free/en/papers/gu-2016-incorporating-copying-mechanism-sequence-sequence/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
7. Kyunghyun Cho. 2016. Noisy parallel approximate decoding for conditional recurrent language model. arXiv preprint arXiv:1605.03835. [find](https://scholar.google.com/scholar?q=Kyunghyun+Cho%2E+2016%2E+Noisy+parallel+approximate+decoding+for+conditional+recurrent+language+model%2E+arXiv+preprint+arXiv%3A1605%2E03835%2E)
8. Jan K. Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. [Attention-based models for speech recognition](https://md.research.elamur.ai/free/en/papers/chorowski-2015-attention-based-models-speech-recognition/). In NIPS, pages 577–585.
9. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. [Deep residual learning for image recognition](https://md.research.elamur.ai/free/en/papers/he-2015-deep-residual-learning-image-recognition/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778.
10. Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear. [find](https://scholar.google.com/scholar?q=Matthew+Honnibal+and+Ines+Montani%2E+2017%2E+spaCy+2%3A+Natural+language+understanding+with+Bloom+embeddings%2C+convolutional+neural+networks+and+incremental+parsing%2E+To+appear%2E)
11. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 201… [find](https://scholar.google.com/scholar?q=Chris+Dyer%2C+Adhiguna+Kuncoro%2C+Miguel+Ballesteros%2C+and+Noah+A%2E+Smith%2E+2016%2E+Recurrent+neural+network+grammars%2E+In+NAACL+HLT+2016%2C+The+2016+Conference+of+the+North+American+Chapter+of+the+Association+fo)
12. Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. [Automatic evaluation of translation quality for distant language pairs](https://md.research.elamur.ai/free/en/papers/isozaki-2010-automatic-evaluation-translation-quality-distant/). In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 944–952. Association for Computational Linguis…
13. Ahmad Emami and Frederick Jelinek. 2005. A neural syntactic language model. Machine Learning, 60(1-3):195–227. [find](https://scholar.google.com/scholar?q=Ahmad+Emami+and+Frederick+Jelinek%2E+2005%2E+A+neural+syntactic+language+model%2E+Machine+Learning%2C+60%281%2D3%29%3A195%E2%80%93227%2E)
14. Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. [Learning to parse and translate improves neural machine translation](https://md.research.elamur.ai/free/en/papers/eriguchi-2017-learning-parse-translate-improves-neural/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 2: Short Papers, pages…
15. Nicolas Ford, Daniel Duckworth, Mohammad Norouzi, and George E. Dahl. 2018. The importance of generation order in language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 -November 4, 2018, pages 2942–2946. [find](https://scholar.google.com/scholar?q=Nicolas+Ford%2C+Daniel+Duckworth%2C+Mohammad+Norouzi%2C+and+George+E%2E+Dahl%2E+2018%2E+The+importance+of+generation+order+in+language+modeling%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natur)
16. Andrej Karpathy and Li Fei-Fei. 2015. [Deep visual-semantic alignments for generating image descriptions](https://md.research.elamur.ai/free/en/papers/karpathy-2014-deep-visual-semantic-alignments-generating/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128–3137.
17. Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. [Deterministic non-autoregressive neural sequence modeling by iterative refinement](https://md.research.elamur.ai/free/en/papers/lee-2018-deterministic-non-autoregressive-neural-sequence/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 -November 4, 2018, pages 1173–1182.
18. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. [Microsoft COCO: Common objects in context](https://md.research.elamur.ai/free/en/papers/lin-2014-microsoft-coco-common-objects-context/). In European Conference on Computer Vision, pages 740–755. Springer.
19. Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. [Non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, Canada, April 30-May 3, 2018, Conference Track Proceedings.
20. Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomás Kociský, Fumin Wang, and Andrew W. Senior. 2016. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin,… [find](https://scholar.google.com/scholar?q=Wang+Ling%2C+Phil+Blunsom%2C+Edward+Grefenstette%2C+Karl+Moritz+Hermann%2C+Toma%CC%81s+Kocisky%CC%81%2C+Fumin+Wang%2C+and+Andrew+W%2E+Senior%2E+2016%2E+Latent+predictor+networks+for+code+generation%2E+In+Proceedings+of+the+54th+An)
21. Shikib Mehri and Leonid Sigal. 2018, Middle-out decoding, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 5523–5534. [find](https://scholar.google.com/scholar?q=Shikib+Mehri+and+Leonid+Sigal%2E+2018%2C+Middle%2Dout+decoding%2C+S%2E+Bengio%2C+H%2E+Wallach%2C+H%2E+Larochelle%2C+K%2E+Grauman%2C+N%2E+Cesa%2DBianchi%2C+and+R%2E+Garnett%2C+editors%2C+Advances+in+Neural+Information+Processing+Systems+)
22. Curran Associates, Inc. and Signal Processing, 2006. [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E+and+Signal+Processing%2C+2006%2E)
23. ICASSP 2006 Proceedings. 2006 IEEE International Conference on, volume 5, pages V–V. IEEE. [find](https://scholar.google.com/scholar?q=ICASSP+2006+Proceedings%2E+2006+IEEE+International+Conference+on%2C+volume+5%2C+pages+V%E2%80%93V%2E+IEEE%2E)
24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [BLEU: A method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
25. Gonzalo Mena, David Belanger, Scott Linderman, and Jasper Snoek. 2018. Learning latent permutations with Gumbel-Sinkhorn networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, Canada, April 30-May 3, 2018, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Gonzalo+Mena%2C+David+Belanger%2C+Scott+Linderman%2C+and+Jasper+Snoek%2E+2018%2E+Learning+latent+permutations+with+Gumbel%2DSinkhorn+networks%2E+In+6th+International+Conference+on+Learning+Representations%2C+ICLR+201)
26. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. [A neural attention model for abstractive sentence summarization](https://md.research.elamur.ai/free/en/papers/rush-2015-neural-attention-model-abstractive-sentence/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
27. Tomáš Mikolov. 2012. Statistical language models based on neural networks. Presentation at Google, Mountain View, 2 April. [find](https://scholar.google.com/scholar?q=Toma%CC%81s%CC%8C+Mikolov%2E+2012%2E+Statistical+language+models+based+on+neural+networks%2E+Presentation+at+Google%2C+Mountain+View%2C+2+April%2E)
28. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. [Neural machine translation of rare words with subword units](https://md.research.elamur.ai/free/en/papers/sennrich-2016-neural-machine-translation-rare-words/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715–1725, Berlin, Germany. Association for Computational Lin…
29. Graham Neubig. 2011. The Kyoto free translation task. [http://www.phontron.com/kftt](http://www.phontron.com/kftt). [↗](http://www.phontron.com/kftt)
30. Yusuke Oda, Hiroyuki Fudaba, Graham Neubig, Hideaki Hata, Sakriani Sakti, Tomoki Toda, and Satoshi Nakamura. 2015. Learning to generate pseudo-code from source code using statistical machine translation. In Proceedings of the 2015 30th IEEE/ACM International Conference on Automated Software Engineer… [find](https://scholar.google.com/scholar?q=Yusuke+Oda%2C+Hiroyuki+Fudaba%2C+Graham+Neubig%2C+Hideaki+Hata%2C+Sakriani+Sakti%2C+Tomoki+Toda%2C+and+Satoshi+Nakamura%2E+2015%2E+Learning+to+generate+pseudo%2Dcode+from+source+code+using+statistical+machine+translati)
31. Aaron van den Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George van den Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, Norman Casagrande, Dominik Grewe, Seb Noury, Sander Dieleman, Erich Elsen, Nal Kalchbrenner, Heiga Zen, Alex Graves, Helen King, To… [find](https://scholar.google.com/scholar?q=Aaron+van+den+Oord%2C+Yazhe+Li%2C+Igor+Babuschkin%2C+Karen+Simonyan%2C+Oriol+Vinyals%2C+Koray+Kavukcuoglu%2C+George+van+den+Driessche%2C+Edward+Lockhart%2C+Luis+Cobo%2C+Florian+Stimberg%2C+Norman+Casagrande%2C+Dominik+Grew)
32. Chris Pal, Charles Sutton, and Andrew McCallum. 2006. Sparse forward-backward using minimum divergence beams for fast training of conditional random fields. In Acoustics, Speech Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. [Self-attention with relative position representations](https://md.research.elamur.ai/free/en/papers/shaw-2018-self-attention-relative-position-representations/). In Proceedin…
33. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. [A study of translation edit rate with targeted human annotation](https://md.research.elamur.ai/free/en/papers/snover-2006-study-translation-edit-rate-targeted/). In Proceedings of Association for Machine Translation in the Americas, pages 223–231.
34. Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.03249. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+William+Chan%2C+Jamie+Kiros%2C+and+Jakob+Uszkoreit%2E+2019%2E+Insertion+transformer%3A+Flexible+sequence+generation+via+insertion+operations%2E+arXiv+preprint+arXiv%3A1902%2E03249%2E)
35. Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems, pages 10107–10116. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+Noam+Shazeer%2C+and+Jakob+Uszkoreit%2E+2018%2E+Blockwise+parallel+decoding+for+deep+autoregressive+models%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+10107%E2%80%9310116%2E)
36. Qing Sun, Stefan Lee, and Dhruv Batra. 2017. Bidirectional beam search: Forward-backward inference in neural sequence models for fill-in-the-blank image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6961–6969. [find](https://scholar.google.com/scholar?q=Qing+Sun%2C+Stefan+Lee%2C+and+Dhruv+Batra%2E+2017%2E+Bidirectional+beam+search%3A+Forward%2Dbackward+inference+in+neural+sequence+models+for+fill%2Din%2Dthe%2Dblank+image+captioning%2E+In+Proceedings+of+the+IEEE+Conferen)
37. Neural Information Processing Systems, pages 2692–2700. [find](https://scholar.google.com/scholar?q=Neural+Information+Processing+Systems%2C+pages+2692%E2%80%932700%2E)
38. Oriol Vinyals and Quoc Le. 2015. [A neural conversational model](https://md.research.elamur.ai/free/en/papers/vinyals-2015-neural-conversational-model/). arXiv preprint arXiv:1506.05869.
39. Chunqi Wang, Ji Zhang, and Haiqing Chen. 2018a. Semi-autoregressive neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 479–488, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Chunqi+Wang%2C+Ji+Zhang%2C+and+Haiqing+Chen%2E+2018a%2E+Semi%2Dautoregressive+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+479%E2%80%93488)
40. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024. [find](https://scholar.google.com/scholar?q=Ilya+Sutskever%2C+James+Martens%2C+and+Geoffrey+E+Hinton%2E+2011%2E+Generating+text+with+recurrent+neural+networks%2E+In+Proceedings+of+the+28th+International+Conference+on+Machine+Learning+%28ICML%2D11%29%2C+pages+101)
41. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104–3112.
42. Xinyi Wang, Hieu Pham, Pengcheng Yin, and Graham Neubig. 2018b. A tree-based decoder for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4772–4777, Brussels, Belgium. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Xinyi+Wang%2C+Hieu+Pham%2C+Pengcheng+Yin%2C+and+Graham+Neubig%2E+2018b%2E+A+tree%2Dbased+decoder+for+neural+machine+translation%2E+In+Proceedings+of+the+2018+Conference+on+Empirical+Methods+in+Natural+Language+Proc)
43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Proceedings of the Annual Conference on Neural Information Processing Systems (NIPS).
44. Sean Welleck, Kianté Brantley, Hal Daumé III, and Kyunghyun Cho. 2019. Non-monotonic sequential text generation. arXiv preprint arXiv:1902.02192. [find](https://scholar.google.com/scholar?q=Sean+Welleck%2C+Kiante%CC%81+Brantley%2C+Hal+Daume%CC%81+III%2C+and+Kyunghyun+Cho%2E+2019%2E+Non%2Dmonotonic+sequential+text+generation%2E+arXiv+preprint+arXiv%3A1902%2E02192%2E)
45. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. [Cider: Consensus-based image description evaluation](https://md.research.elamur.ai/free/en/papers/vedantam-2014-cider-consensus-based-image-description/). In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4566–4575.
46. Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2016. [Diverse beam search: Decoding diverse solutions from neural sequence models](https://md.research.elamur.ai/free/en/papers/vijayakumar-2016-diverse-beam-search-decoding-diverse/). arXiv preprint arXiv:1610.02424.
47. Lijun Wu, Xu Tan, Di He, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018. Beyond error propagation in neural machine translation: Characteristics of language also matter. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3602–3611, Brussels, Bel… [find](https://scholar.google.com/scholar?q=Lijun+Wu%2C+Xu+Tan%2C+Di+He%2C+Fei+Tian%2C+Tao+Qin%2C+Jianhuang+Lai%2C+and+Tie%2DYan+Liu%2E+2018%2E+Beyond+error+propagation+in+neural+machine+translation%3A+Characteristics+of+language+also+matter%2E+In+Proceedings+of+the)
48. Kenji Yamada and Kevin Knight. 2001. [A syntax-based statistical translation model](https://md.research.elamur.ai/free/en/papers/yamada-2001-syntax-based-statistical-translation-model/). In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics.
49. Pengcheng Yin and Graham Neubig. 2017. [A syntactic neural model for general-purpose code generation](https://md.research.elamur.ai/free/en/papers/yin-2017-syntactic-neural-model-general-purpose/). In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440–450, Vancouver, Canada. Association for Computational Linguistics.
50. Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. 2016. Order matters: Sequence to sequence for sets. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Samy+Bengio%2C+and+Manjunath+Kudlur%2E+2016%2E+Order+matters%3A+Sequence+to+sequence+for+sets%2E+In+4th+International+Conference+on+Learning+Representations%2C+ICLR+2016%2C+San+Juan%2C+Puerto+Rico%2C+May)
51. Xingxing Zhang, Liang Lu, and Mirella Lapata. 2016. Top-down tree long short-term memory networks. In Proceedings of the 2016 [find](https://scholar.google.com/scholar?q=Xingxing+Zhang%2C+Liang+Lu%2C+and+Mirella+Lapata%2E+2016%2E+Top%2Ddown+tree+long+short%2Dterm+memory+networks%2E+In+Proceedings+of+the+2016)
52. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 310–320, San Diego, California. Association for Compu tational Linguistics. transla… [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Meire+Fortunato%2C+and+Navdeep+Jaitly%2E+2015%2E+Pointer+networks%2E+In+Advances+in+Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics%3A+Human+Language+Tec)
53. Long Zhou, Jiajun Zhang, Chengqing Zong, and Heng Yu. 2019b. Sequence generation: From both sides to the middle. IJCAI. [find](https://scholar.google.com/scholar?q=Long+Zhou%2C+Jiajun+Zhang%2C+Chengqing+Zong%2C+and+Heng+Yu%2E+2019b%2E+Sequence+generation%3A+From+both+sides+to+the+middle%2E+IJCAI%2E)
54. Wanrong Zhu, Zhiting Hu, and Eric Xing. 2019. Text Infilling. arXiv, arXiv:1901.00158. — [arXiv:1901.00158](https://arxiv.org/abs/1901.00158) [find](https://scholar.google.com/scholar?q=Wanrong+Zhu%2C+Zhiting+Hu%2C+and+Eric+Xing%2E+2019%2E+Text+Infilling%2E+arXiv%2C+arXiv%3A1901%2E00158%2E+%E2%80%94+%5BarXiv%3A1901%2E00158%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1901%2E00158%29)
55. Long Zhou, Jiajun Zhang, and Chengqing Zong. 2019a. Synchronous bidirectional neural machine [find](https://scholar.google.com/scholar?q=Long+Zhou%2C+Jiajun+Zhang%2C+and+Chengqing+Zong%2E+2019a%2E+Synchronous+bidirectional+neural+machine)

*Updated 29 October 2026*