---
title: "Order Matters: Sequence to sequence for sets (2015)"
description: "For unordered inputs, they introduce the Read-Process-and-Write architecture. 5.0 BLEU. 23 references, cited by 89 papers here."
authors: ["Oriol Vinyals", "Samy Bengio", "Manjunath Kudlur"]
year: 2015
venue: "arXiv:1511.06391"
source: "https://arxiv.org/abs/1511.06391"
doi: "10.48550/arxiv.1511.06391"
arxiv: "1511.06391"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/vinyals-2015-order-matters-sequence-sequence-sets/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/vinyals-2015-order-matters-sequence-sequence-sets/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Order Matters: Sequence to sequence for sets

# Order Matters: Sequence to sequence for sets

**Authors:** [Oriol Vinyals](https://md.research.elamur.ai/free/en/authors/oriol-vinyals/), [Samy Bengio](https://md.research.elamur.ai/free/en/authors/samy-bengio/) and [Manjunath Kudlur](https://md.research.elamur.ai/free/en/authors/manjunath-kudlur/) · **Year:** 2015 · **Venue:** arXiv:1511.06391 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1511.06391) · [PDF](https://arxiv.org/pdf/1511.06391) · [DOI 10.48550/arxiv.1511.06391](https://doi.org/10.48550/arxiv.1511.06391) · [arXiv 1511.06391](https://arxiv.org/abs/1511.06391) · [OpenAlex](https://openalex.org/W2173183968)

**Citations:** 156 citations (OpenAlex, 2026-10-14) · [89](https://md.research.elamur.ai/free/en/papers/vinyals-2015-order-matters-sequence-sequence-sets/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Seq2seq models assume that inputs and outputs arrive as ordered sequences. Many problems, such as sorting a collection of numbers or predicting the joint distribution of random variables, involve data with no obvious order. The authors show that the chosen ordering can strongly affect how well such models learn, even when no natural ordering exists. They attribute this to non-convex optimization and to how well an ordering fits the model's prior.

For unordered inputs, they introduce the Read-Process-and-Write architecture. It stores set elements in a memory and reads them through content-based attention driven by an LSTM, so the encoding does not change when elements are permuted and grows with the size of the set. For unordered outputs, they propose a training loss that searches over candidate output orderings rather than fixing one in advance.

Experiments cover sorting, graphical-model estimation, language modeling and parsing. In a 5-gram task, the model finds a good ordering on its own. The work matters for any pipeline that must linearize sets or structures before feeding them to recurrent networks.

## Key points
- Earlier work showed order effects: reversing source sentences gave Sutskever et al. (2014) a 5.0 BLEU gain in translation, and gave roughly 0.5% absolute F1 in constituency parsing.
- Sorting points by angle before convex hull computation with Pointer Networks made training faster and raised accuracy by up to 10% absolute on the hardest instances.
- Read-Process-and-Write combines an attention-based memory with an LSTM "process" block, yielding permutation-invariant set representations. It can be viewed as a special case of Memory Networks or Neural Turing Machines.
- In the 5-gram language modeling task, the natural order reached a validation perplexity of 225, while a fixed shuffled order (5,1,3,4,2) gave 280. When searching over 2 orders or over all 5! orders, the model still converged to 225.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Google](https://md.research.elamur.ai/free/en/companies/google/) — Company, Mountain View, United States · 492 works · 440,463 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Oriol Vinyals, Samy Bengio and Manjunath Kudlur. 2015. Order Matters: Sequence to sequence for sets. arXiv:1511.06391.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{vinyals2015order,
  title = {{Order Matters: Sequence to sequence for sets}},
  author = {Oriol Vinyals, Samy Bengio and Manjunath Kudlur},
  year = {2015},
  doi = {10.48550/arxiv.1511.06391},
  eprint = {1511.06391},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1511.06391}
}
```

- [← previous Entity-Centric Coreference Resolution with Model Stacking](https://md.research.elamur.ai/free/en/papers/clark-2015-entity-centric-coreference-resolution-model/)
- [next → Demographic Factors Improve Classification Performance](https://md.research.elamur.ai/free/en/papers/hovy-2015-demographic-factors-improve-classification-performance/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bahdanau, D., Cho, K., and Bengio, Y. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). In Proc. ICLR, 2015a.
2. Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., and Bengio, Y. End-to-end attention-based large vocabulary speech recognition. arXiv preprint arXiv:1508.04395, 2015b. [find](https://scholar.google.com/scholar?q=Bahdanau%2C+D%2E%2C+Chorowski%2C+J%2E%2C+Serdyuk%2C+D%2E%2C+Brakel%2C+P%2E%2C+and+Bengio%2C+Y%2E+End%2Dto%2Dend+attention%2Dbased+large+vocabulary+speech+recognition%2E+arXiv+preprint+arXiv%3A1508%2E04395%2C+2015b%2E)
3. Bakir, G., Hofmann, T., Scholkopf, B., Smola, A. J., Taskar, B., and Vishwanathan, S.V.N. (eds.). Predicting Structured Data. MIT Press, 2007. [find](https://scholar.google.com/scholar?q=Bakir%2C+G%2E%2C+Hofmann%2C+T%2E%2C+Scholkopf%2C+B%2E%2C+Smola%2C+A%2E+J%2E%2C+Taskar%2C+B%2E%2C+and+Vishwanathan%2C+S%2EV%2EN%2E+%28eds%2E%29%2E+Predicting+Structured+Data%2E+MIT+Press%2C+2007%2E)
4. Chan, W., Jaitly, N., Le, Q. V., and Vinyals, O. Listen, attend and spell. arXiv, abs/1508.01211, 2015. URL http://arxiv.org/abs/1508.01211. [↗](http://arxiv.org/abs/1508.01211)
5. Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. EMNLP, 2014. — [Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation](https://md.research.elamur.ai/free/en/papers/cho-2014-learning-phrase-representations-rnn-encoder/)
6. Donahue, J., Hendricks, L. A., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., and Darrell, T. [Long-term recurrent convolutional networks for visual recognition and description](https://md.research.elamur.ai/free/en/papers/donahue-2014-long-term-recurrent-convolutional-networks/). In Proc. CVPR, 2015.
7. Graves, A. Supervised Sequence Labelling with Recurrent Neural Networks. Springer, 2012. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E+Supervised+Sequence+Labelling+with+Recurrent+Neural+Networks%2E+Springer%2C+2012%2E)
8. Graves, A., Wayne, G., and Danihelka, I. Neural turing machines. In arXiv preprint arXiv:1410.5401, 2014. [find](https://scholar.google.com/scholar?q=Graves%2C+A%2E%2C+Wayne%2C+G%2E%2C+and+Danihelka%2C+I%2E+Neural+turing+machines%2E+In+arXiv+preprint+arXiv%3A1410%2E5401%2C+2014%2E)
9. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T. N., and Kingsbury, B. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29:82–97, 2012. [find](https://scholar.google.com/scholar?q=Hinton%2C+G%2E%2C+Deng%2C+L%2E%2C+Yu%2C+D%2E%2C+Dahl%2C+G%2E%2C+Mohamed%2C+A%2E%2C+Jaitly%2C+N%2E%2C+Senior%2C+A%2E%2C+Vanhoucke%2C+V%2E%2C+Nguyen%2C+P%2E%2C+Sainath%2C+T%2E+N%2E%2C+and+Kingsbury%2C+B%2E+Deep+neural+networks+for+acoustic+modeling+in+speech+recogniti)
10. Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural Computation, 9(8), 1997. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+and+Schmidhuber%2C+J%2E+Long+short%2Dterm+memory%2E+Neural+Computation%2C+9%288%29%2C+1997%2E)
11. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, ICML, 2015. [find](https://scholar.google.com/scholar?q=Ioffe%2C+S%2E+and+Szegedy%2C+C%2E+Batch+normalization%3A+Accelerating+deep+network+training+by+reducing+internal+covariate+shift%2E+In+Proceedings+of+the+32nd+International+Conference+on+Machine+Learning%2C+ICML%2C+2)
12. Kalchbrenner, N. and Blunsom, P. [Recurrent continuous translation models](https://md.research.elamur.ai/free/en/papers/kalchbrenner-2013-recurrent-continuous-translation-models/). In Proc. EMNLP, 2013.
13. Maas, A. L., Miller, S. D., O’Neil, T. M., and Ng, A. Y. Word-level acoustic modeling with convolutional vector regression. In ICML 2012 Workshop on Representation Learning, 2012. [find](https://scholar.google.com/scholar?q=Maas%2C+A%2E+L%2E%2C+Miller%2C+S%2E+D%2E%2C+O%E2%80%99Neil%2C+T%2E+M%2E%2C+and+Ng%2C+A%2E+Y%2E+Word%2Dlevel+acoustic+modeling+with+convolutional+vector+regression%2E+In+ICML+2012+Workshop+on+Representation+Learning%2C+2012%2E)
14. Mao, J., Xu, W., Yang, Y., Wang, J., Huang, Z., and Yuille, A. L. [Deep captioning with multimodal recurrent neural networks (m-RNN)](https://md.research.elamur.ai/free/en/papers/mao-2014-deep-captioning-multimodal-recurrent-neural/). In International Conference on Learning Representations, 2015.
15. Socher, R., Manning, C. D., and Ng, A. Y. Learning continuous phrase representations and syntactic parsing with recursive neural networks. In Advances in Neural Information Processing Systems, 2010. [find](https://scholar.google.com/scholar?q=Socher%2C+R%2E%2C+Manning%2C+C%2E+D%2E%2C+and+Ng%2C+A%2E+Y%2E+Learning+continuous+phrase+representations+and+syntactic+parsing+with+recursive+neural+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2010%2E)
16. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Proc. NIPS, 2014.
17. Vinyals, O., Fortunato, M., and Jaitly, N. Pointer networks. In Advances in Neural Information Processing Systems, NIPS, 2015a. [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E%2C+Fortunato%2C+M%2E%2C+and+Jaitly%2C+N%2E+Pointer+networks%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+NIPS%2C+2015a%2E)
18. Vinyals, O., Kaiser, L., Koo, T., Petrov, S., Sutskever, I., and Hinton, G. Grammar as a foreign language. In Advances in Neural Information Processing Systems, 2015b. [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E%2C+Kaiser%2C+L%2E%2C+Koo%2C+T%2E%2C+Petrov%2C+S%2E%2C+Sutskever%2C+I%2E%2C+and+Hinton%2C+G%2E+Grammar+as+a+foreign+language%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+2015b%2E)
19. Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. [Show and tell: A neural image caption generator](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/). In Proc. CVPR, 2015c.
20. Weston, J., Chopra, S., and Bordes, A. Memory networks. In International Conference on Learning Representations, ICLR, 2015. [find](https://scholar.google.com/scholar?q=Weston%2C+J%2E%2C+Chopra%2C+S%2E%2C+and+Bordes%2C+A%2E+Memory+networks%2E+In+International+Conference+on+Learning+Representations%2C+ICLR%2C+2015%2E)
21. Zaremba, W. and Sutskever, I. Learning to execute. arXiv, abs/1410.4615, 2014. [find](https://scholar.google.com/scholar?q=Zaremba%2C+W%2E+and+Sutskever%2C+I%2E+Learning+to+execute%2E+arXiv%2C+abs%2F1410%2E4615%2C+2014%2E)
22. Zaremba, W. and Sutskever, I. Reinforcement learning neural turing machines. arXiv, abs/1505.[00521, 2015](https://md.research.elamur.ai/free/en/papers/zaremba-2015-reinforcement-learning-neural-turing-machines/).
23. Zaremba, W., Sutskever, I., and Vinyals, O. [Recurrent neural network regularization](https://md.research.elamur.ai/free/en/papers/zaremba-2014-recurrent-neural-network-regularization/). arXiv, abs/1409.2329, 2014.

*Updated 2 November 2026*