---
title: "FlowSeq — paper by Ma et al. (2019) — summary and ideas"
description: "Autoregressive seq2seq models produce output one token after another, which limits parallel… 32 references, cited by 72 papers here."
authors: ["Xuezhe Ma", "Chunting Zhou", "Xian Li", "Graham Neubig", "Eduard Hovy"]
year: 2019
venue: "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)"
source: "https://aclanthology.org/D19-1437/"
doi: "10.18653/v1/d19-1437"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ma-2019-flowseq-non-autoregressive-conditional-sequence/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ma-2019-flowseq-non-autoregressive-conditional-sequence/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/) / FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

# FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

**Authors:** [Xuezhe Ma](https://md.research.elamur.ai/free/en/authors/xuezhe-ma/), [Chunting Zhou](https://md.research.elamur.ai/free/en/authors/chunting-zhou/), [Xian Li](https://md.research.elamur.ai/free/en/authors/xian-li/), [Graham Neubig](https://md.research.elamur.ai/free/en/authors/graham-neubig/) and [Eduard Hovy](https://md.research.elamur.ai/free/en/authors/eduard-hovy/) · **Year:** 2019 · **Venue:** [Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2019](https://md.research.elamur.ai/free/en/venues/emnlp/2019/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-1437/) · [PDF](https://aclanthology.org/D19-1437.pdf) · [DOI 10.18653/v1/d19-1437](https://doi.org/10.18653/v1/d19-1437) · [OpenAlex](https://openalex.org/W2970832665)

**Citations:** 142 citations (OpenAlex, 2026-10-14) · [72](https://md.research.elamur.ai/free/en/papers/ma-2019-flowseq-non-autoregressive-conditional-sequence/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
Autoregressive seq2seq models produce output one token after another, which limits parallel decoding on GPUs. Non-autoregressive models predict all positions at once, but they usually lose accuracy because they struggle to capture how output words depend on each other. FlowSeq addresses this with a sequence of continuous latent vectors, one per target position. The prior over these vectors, conditioned on the source, is modeled with a generative flow adapted from Glow. The model is trained variationally with a diagonal-Gaussian posterior network, and the decoder then predicts every target token independently, without causal masking. The authors evaluated it on the WMT2014, WMT2016 and IWSLT-2014 translation benchmarks. FlowSeq performs on par with leading non-autoregressive systems and stays only slightly behind the contemporaneous CMLM. Its decoding time stays nearly flat as sentences get longer, while a Transformer's grows. The work matters because it is, by the authors' account, the first non-autoregressive seq2seq model built on generative flows. It also shows that an expressive latent prior can recover much of the quality lost by parallel generation.

## Key points
- Each step of the prior flow stacks three invertible components: actnorm, an invertible multi-head linear layer, and a coupling layer. These steps are arranged in a Glow-style multi-scale architecture.
- Two training techniques are used. Token-level dropout in the posterior stops latents from simply copying their own target token. Zero-initializing the final layers that produce the posterior mean and log-variance stabilizes very deep flows.
- With a batch size of 128 on a single NVIDIA TITAN X (WMT14 EN-DE test set), FlowSeq decoding sped up by 594% for the base model and 403% for the large model. Transformer with beam search gained less from batching.
- NPD and IWD rescoring used reduced-temperature sampling, with temperatures from 0.1 to 1.0. The number of samples per length mattered most, and temperatures of 0.3–0.5 worked best.
- A diversity study on a WMT14 EN-DE subset with ten references used pairwise-BLEU and leave-one-out BLEU. Raising the sampling temperature made outputs more diverse with almost no loss in quality, though FlowSeq still trails autoregressive generation. Code is published at github.com/XuezheMax/flowseq.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [Meta](https://md.research.elamur.ai/free/en/companies/meta/) — Company, Menlo Park, United States · 158 works · 211,459 citations
- [Meta (United Kingdom)](https://md.research.elamur.ai/free/en/companies/meta-united-kingdom/) — Company, London, United Kingdom · 39 works · 11,541 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works

## Cite

Cite as: Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig and Eduard Hovy. 2019. FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ma2019flowseq,
  title = {{FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow}},
  author = {Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig and Eduard Hovy},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1437},
  url = {https://doi.org/10.18653/v1/d19-1437}
}
```

- [← previous Probing the Need for Visual Context in Multimodal Machine Translation](https://md.research.elamur.ai/free/en/papers/caglayan-2019-probing-need-visual-context-multimodal/)
- [next → Extractive Summarization of Long Documents by Combining Global and Local Context](https://md.research.elamur.ai/free/en/papers/xiao-2019-extractive-summarization-long-documents-combining/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/4/)

## References

32 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In International Conference on Learning Representations (ICLR).
2. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. [Generating sentences from a continuous space](https://md.research.elamur.ai/free/en/papers/bowman-2016-generating-sentences-continuous-space/). arXiv preprint arXiv:1511.06349.
3. Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2015. Importance weighted autoencoders. arXiv preprint arXiv:1509.00519. — [arXiv:1509.00519](https://arxiv.org/abs/1509.00519) [find](https://scholar.google.com/scholar?q=Yuri+Burda%2C+Roger+Grosse%2C+and+Ruslan+Salakhutdinov%2E+2015%2E+Importance+weighted+autoencoders%2E+arXiv+preprint+arXiv%3A1509%2E00519%2E+%E2%80%94+%5BarXiv%3A1509%2E00519%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1509%2E00519%29)
4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. [Bert: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). arXiv preprint arXiv:1810.04805.
5. Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. 2016. Density estimation using real nvp. arXiv preprint arXiv:1605.08803. — [arXiv:1605.08803](https://arxiv.org/abs/1605.08803) [find](https://scholar.google.com/scholar?q=Laurent+Dinh%2C+Jascha+Sohl%2DDickstein%2C+and+Samy+Bengio%2E+2016%2E+Density+estimation+using+real+nvp%2E+arXiv+preprint+arXiv%3A1605%2E08803%2E+%E2%80%94+%5BarXiv%3A1605%2E08803%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1605%2E08803%29)
6. Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Constant-time machine translation with conditional masked language models. arXiv preprint arXiv:1904.09324. — [arXiv:1904.09324](https://arxiv.org/abs/1904.09324) [find](https://scholar.google.com/scholar?q=Marjan+Ghazvininejad%2C+Omer+Levy%2C+Yinhan+Liu%2C+and+Luke+Zettlemoyer%2E+2019%2E+Constant%2Dtime+machine+translation+with+conditional+masked+language+models%2E+arXiv+preprint+arXiv%3A1904%2E09324%2E+%E2%80%94+%5BarXiv%3A1904%2E09324)
7. Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. [Non-autoregressive neural machine translation](https://md.research.elamur.ai/free/en/papers/gu-2017-non-autoregressive-neural-machine-translation/). In International Conference on Learning Representations (ICLR).
8. Jiatao Gu, Qi Liu, and Kyunghyun Cho. 2019. Insertion-based decoding with automatically inferred generation order. arXiv preprint arXiv:1902.01370. [find](https://scholar.google.com/scholar?q=Jiatao+Gu%2C+Qi+Liu%2C+and+Kyunghyun+Cho%2E+2019%2E+Insertion%2Dbased+decoding+with+automatically+inferred+generation+order%2E+arXiv+preprint+arXiv%3A1902%2E01370%2E)
9. Sergey Ioffe and Christian Szegedy. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pages 448–456. [find](https://scholar.google.com/scholar?q=Sergey+Ioffe+and+Christian+Szegedy%2E+2015%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+International+Conference+on+Machine+Learning%2C+pages+448%E2%80%93456%2E)
10. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma+and+Jimmy+Ba%2E+2014%2E+Adam%3A+A+method+for+stochastic+optimization%2E+arXiv+preprint+arXiv%3A1412%2E6980%2E)
11. Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improving variational inference with inverse autoregressive flow. The 29th Conference on Neural Information Processing Systems. [find](https://scholar.google.com/scholar?q=Diederik+P+Kingma%2C+Tim+Salimans%2C+Rafal+Jozefowicz%2C+Xi+Chen%2C+Ilya+Sutskever%2C+and+Max+Welling%2E+2016%2E+Improving+variational+inference+with+inverse+autoregressive+flow%2E+The+29th+Conference+on+Neural+Infor)
12. Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1x1 convolutions. In Advances in Neural Information Processing Systems, pages 10215–10224. [find](https://scholar.google.com/scholar?q=Durk+P+Kingma+and+Prafulla+Dhariwal%2E+2018%2E+Glow%3A+Generative+flow+with+invertible+1x1+convolutions%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+10215%E2%80%9310224%2E)
13. Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. [Deterministic non-autoregressive neural sequence modeling by iterative refinement](https://md.research.elamur.ai/free/en/papers/lee-2018-deterministic-non-autoregressive-neural-sequence/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1173– 1182.
14. Jindřich Libovickỳ and Jindřich Helcl. 2018. [End-to-end non-autoregressive neural machine translation with connectionist temporal classification](https://md.research.elamur.ai/free/en/papers/libovicky-2018-end-end-non-autoregressive-neural/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016– 3021.
15. Xuezhe Ma and Eduard Hovy. 2019. Macow: Masked convolutional generative flow. arXiv preprint arXiv:1902.04208. — [arXiv:1902.04208](https://arxiv.org/abs/1902.04208) [find](https://scholar.google.com/scholar?q=Xuezhe+Ma+and+Eduard+Hovy%2E+2019%2E+Macow%3A+Masked+convolutional+generative+flow%2E+arXiv+preprint+arXiv%3A1902%2E04208%2E+%E2%80%94+%5BarXiv%3A1902%2E04208%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1902%2E04208%29)
16. Xuezhe Ma, Zecong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stack-pointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403–1414. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma%2C+Zecong+Hu%2C+Jingzhou+Liu%2C+Nanyun+Peng%2C+Graham+Neubig%2C+and+Eduard+Hovy%2E+2018%2E+Stack%2Dpointer+networks+for+dependency+parsing%2E+In+Proceedings+of+the+56th+Annual+Meeting+of+the+Association+for+C)
17. Xuezhe Ma, Chunting Zhou, and Eduard Hovy. 2019. Mae: Mutual posterior-divergence regularization for variational autoencoders. In Proceedings of the 7th International Conference on Learning Representations (ICLR-2019), New Orleans, Louisiana, USA. [find](https://scholar.google.com/scholar?q=Xuezhe+Ma%2C+Chunting+Zhou%2C+and+Eduard+Hovy%2E+2019%2E+Mae%3A+Mutual+posterior%2Ddivergence+regularization+for+variational+autoencoders%2E+In+Proceedings+of+the+7th+International+Conference+on+Learning+Representa)
18. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. [context2vec: Learning generic context embedding with bidirectional LSTM](https://md.research.elamur.ai/free/en/papers/melamud-2016-context2vec-learning-generic-context-embedding/). In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 51–61, Berlin, Germany. Association for Computational Linguistics.
19. Myle Ott, Michael Auli, David Grangier, et al. 2018. [Analyzing uncertainty in neural machine translation](https://md.research.elamur.ai/free/en/papers/ott-2018-analyzing-uncertainty-neural-machine-translation/). In International Conference on Machine Learning, pages 3953–3962.
20. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. [fairseq: A fast, extensible toolkit for sequence modeling](https://md.research.elamur.ai/free/en/papers/ott-2019-fairseq-fast-extensible-toolkit-sequence/). In Proceedings of NAACL-HLT 2019: Demonstrations.
21. George Papamakarios, Theo Pavlakou, and Iain Murray. 2017. Masked autoregressive flow for density estimation. In Advances in Neural Information Processing Systems, pages 2338–2347. [find](https://scholar.google.com/scholar?q=George+Papamakarios%2C+Theo+Pavlakou%2C+and+Iain+Murray%2E+2017%2E+Masked+autoregressive+flow+for+density+estimation%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+pages+2338%E2%80%932347%2E)
22. Ryan Prenger, Rafael Valle, and Bryan Catanzaro. 2019. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 3617–3621. IEEE. [find](https://scholar.google.com/scholar?q=Ryan+Prenger%2C+Rafael+Valle%2C+and+Bryan+Catanzaro%2E+2019%2E+Waveglow%3A+A+flow%2Dbased+generative+network+for+speech+synthesis%2E+In+ICASSP+2019%2D2019+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal)
23. Danilo Jimenez Rezende and Shakir Mohamed. 2015. Variational inference with normalizing flows. In Proceedings of the 32nd International Conference on International Conference on Machine Learning-Volume 37, pages 1530–1538. JMLR. org. [find](https://scholar.google.com/scholar?q=Danilo+Jimenez+Rezende+and+Shakir+Mohamed%2E+2015%2E+Variational+inference+with+normalizing+flows%2E+In+Proceedings+of+the+32nd+International+Conference+on+International+Conference+on+Machine+Learning%2DVolum)
24. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389. [find](https://scholar.google.com/scholar?q=Alexander+M+Rush%2C+Sumit+Chopra%2C+and+Jason+Weston%2E+2015%2E+A+neural+attention+model+for+abstractive+sentence+summarization%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+)
25. Tianxiao Shen, Myle Ott, Michael Auli, et al. 2019. Mixture models for diverse machine translation: Tricks of the trade. In International Conference on Machine Learning, pages 5719–5728. [find](https://scholar.google.com/scholar?q=Tianxiao+Shen%2C+Myle+Ott%2C+Michael+Auli%2C+et+al%2E+2019%2E+Mixture+models+for+diverse+machine+translation%3A+Tricks+of+the+trade%2E+In+International+Conference+on+Machine+Learning%2C+pages+5719%E2%80%935728%2E)
26. Raphael Shu, Jason Lee, Hideki Nakayama, and Kyunghyun Cho. 2019. Latent-variable non-autoregressive neural machine translation with deterministic inference using a delta posterior. arXiv preprint arXiv:1908.07181. — [arXiv:1908.07181](https://arxiv.org/abs/1908.07181) [find](https://scholar.google.com/scholar?q=Raphael+Shu%2C+Jason+Lee%2C+Hideki+Nakayama%2C+and+Kyunghyun+Cho%2E+2019%2E+Latent%2Dvariable+non%2Dautoregressive+neural+machine+translation+with+deterministic+inference+using+a+delta+posterior%2E+arXiv+preprint+arX)
27. Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible sequence generation via insertion operations. arXiv preprint arXiv:1902.03249. [find](https://scholar.google.com/scholar?q=Mitchell+Stern%2C+William+Chan%2C+Jamie+Kiros%2C+and+Jakob+Uszkoreit%2E+2019%2E+Insertion+transformer%3A+Flexible+sequence+generation+via+insertion+operations%2E+arXiv+preprint+arXiv%3A1902%2E03249%2E)
28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. [Attention is all you need](https://md.research.elamur.ai/free/en/papers/vaswani-2017-attention-all-need/). In Advances in neural information processing systems, pages 5998–6008.
29. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164.
30. Martin J Wainwright, Michael I Jordan, et al. 2008. Graphical models, exponential families, and variational inference. Foundations and Trends R in Machine Learning, 1(1–2):1–305. [find](https://scholar.google.com/scholar?q=Martin+J+Wainwright%2C+Michael+I+Jordan%2C+et+al%2E+2008%2E+Graphical+models%2C+exponential+families%2C+and+variational+inference%2E+Foundations+and+Trends+R+in+Machine+Learning%2C+1%281%E2%80%932%29%3A1%E2%80%93305%2E)
31. Yiren Wang, Fei Tian, Di He, Tao Qin, ChengXiang Zhai, and Tie-Yan Liu. 2019. Non-autoregressive machine translation with auxiliary regularization. arXiv preprint arXiv:1902.10245. — [arXiv:1902.10245](https://arxiv.org/abs/1902.10245) [find](https://scholar.google.com/scholar?q=Yiren+Wang%2C+Fei+Tian%2C+Di+He%2C+Tao+Qin%2C+ChengXiang+Zhai%2C+and+Tie%2DYan+Liu%2E+2019%2E+Non%2Dautoregressive+machine+translation+with+auxiliary+regularization%2E+arXiv+preprint+arXiv%3A1902%2E10245%2E+%E2%80%94+%5BarXiv%3A1902%2E10245)
32. Zachary Ziegler and Alexander Rush. 2019. Latent normalizing flows for discrete sequences. In International Conference on Machine Learning, pages 7673–7682. [find](https://scholar.google.com/scholar?q=Zachary+Ziegler+and+Alexander+Rush%2E+2019%2E+Latent+normalizing+flows+for+discrete+sequences%2E+In+International+Conference+on+Machine+Learning%2C+pages+7673%E2%80%937682%2E)

*Updated 24 October 2026*