---
title: "A Hierarchical Neural Autoencoder for Paragraphs and Documents"
description: "They compare three LSTM designs. 28 references, cited by 118 papers here. Among them: 169."
authors: ["Jiwei Li", "Minh-Thang Luong", "Dan Jurafsky"]
year: 2015
venue: "arXiv:1506.01057"
source: "https://arxiv.org/abs/1506.01057"
doi: "10.48550/arxiv.1506.01057"
arxiv: "1506.01057"
licence: "arXiv — non-exclusive license to distribute"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2015-hierarchical-neural-autoencoder-paragraphs-documents/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2015-hierarchical-neural-autoencoder-paragraphs-documents/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / A Hierarchical Neural Autoencoder for Paragraphs and Documents

# A Hierarchical Neural Autoencoder for Paragraphs and Documents

**Authors:** [Jiwei Li](https://md.research.elamur.ai/free/en/authors/jiwei-li/), [Minh-Thang Luong](https://md.research.elamur.ai/free/en/authors/minh-thang-luong/) and [Dan Jurafsky](https://md.research.elamur.ai/free/en/authors/dan-jurafsky/) · **Year:** 2015 · **Venue:** arXiv:1506.01057 · **Licence:** [arXiv — non-exclusive license to distribute](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/1506.01057) · [PDF](https://arxiv.org/pdf/1506.01057) · [DOI 10.48550/arxiv.1506.01057](https://doi.org/10.48550/arxiv.1506.01057) · [arXiv 1506.01057](https://arxiv.org/abs/1506.01057) · [OpenAlex](https://openalex.org/W2950752421)

**Citations:** 169 citations (OpenAlex, 2026-10-14) · [118](https://md.research.elamur.ai/free/en/papers/li-2015-hierarchical-neural-autoencoder-paragraphs-documents/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
The paper asks whether recurrent networks, which already handle single sentences well, can also capture how sentences combine into a coherent paragraph or document. As a first step toward multi-sentence generation, the authors frame the task as autoencoding: compress a whole paragraph into one vector, then reconstruct the original text from it. They compare three LSTM designs. The first is a flat sequence-to-sequence baseline that ignores sentence boundaries. The second is a hierarchical model, in which a word-level LSTM builds sentence vectors and a sentence-level LSTM builds a document vector, with the decoder mirroring this structure. The third adds sentence-level attention to the hierarchical model. Experiments use a hotel-review corpus and Wikipedia text. Outputs are scored with BLEU, ROUGE, and a custom order-based coherence measure. The hierarchical variants consistently outperform the flat model, and attention brings further gains. This suggests that neural encoders can retain discourse-level information, which is relevant for summarization, question answering, and dialogue generation.

## Key points
- Three models are compared: Standard LSTM, Hierarchical LSTM, and Hierarchical LSTM with attention. The hierarchical decoder stops when it predicts a special end-of-document token.
- On Hotel Review, Hierarchical+Attention reaches BLEU 0.285, ROUGE-1 0.624, and ROUGE-2 0.355, compared with 0.241, 0.571, and 0.302 for the Standard model.
- On Wikipedia, scores are lower across the board, because open-domain text is less formulaic. The best model reaches BLEU 0.220, ROUGE-1 0.544, and ROUGE-2 0.291.
- The coherence score L measures how far sentence order drifts from the input, so lower is better. Hierarchical+Attention gets 1.57 on hotel reviews (Standard: 1.92) and 2.04 on Wikipedia (Standard: 2.75).
- Code was released by the authors on Jiwei Li's Stanford page.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 506 works · 330,053 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

## Cite

Cite as: Jiwei Li, Minh-Thang Luong and Dan Jurafsky. 2015. A Hierarchical Neural Autoencoder for Paragraphs and Documents. arXiv:1506.01057.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2015hierarchical,
  title = {{A Hierarchical Neural Autoencoder for Paragraphs and Documents}},
  author = {Jiwei Li, Minh-Thang Luong and Dan Jurafsky},
  year = {2015},
  doi = {10.48550/arxiv.1506.01057},
  eprint = {1506.01057},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1506.01057}
}
```

- [← previous Show Me Your Evidence - an Automatic Method for Context Dependent Evidence Detection](https://md.research.elamur.ai/free/en/papers/rinott-2015-show-me-your-evidence-automatic/)
- [next → Shared Tasks of the 2015 Workshop on Noisy User-generated Text: Twitter Lexical Normalization and Named Entity Recognition](https://md.research.elamur.ai/free/en/papers/baldwin-2015-shared-tasks-2015-workshop-noisy/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

28 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. [Neural machine translation by jointly learning to align and translate](https://md.research.elamur.ai/free/en/papers/bahdanau-2014-neural-machine-translation-jointly-learning/). arXiv preprint arXiv:1409.0473.
2. Regina Barzilay and Mirella Lapata. 2008. [Modeling local coherence: An entity-based approach](https://md.research.elamur.ai/free/en/papers/barzilay-2008-modeling-local-coherence-entity-based/). Computational Linguistics, 34(1):1–34.
3. Regina Barzilay and Lillian Lee. 2004. [Catching the drift: Probabilistic content models, with applications to generation and summarization](https://md.research.elamur.ai/free/en/papers/barzilay-2004-catching-drift-probabilistic-content-models/). arXiv preprint cs/0405039.
4. Asli Celikyilmaz and Dilek Hakkani-Tür. 2011. Discovery of topically coherent sentences for extractive summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 491–499. Association for Computational Lingui… [find](https://scholar.google.com/scholar?q=Asli+Celikyilmaz+and+Dilek+Hakkani%2DTu%CC%88r%2E+2011%2E+Discovery+of+topically+coherent+sentences+for+extractive+summarization%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Li)
5. Micha Elsner and Eugene Charniak. [Coreference-inspired coherence modeling](https://md.research.elamur.ai/free/en/papers/elsner-2008-coreference-inspired-coherence-modeling/). In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pages 41–44. Association for Computational Linguistics.
6. Vanessa Wei Feng and Graeme Hirst. 2012. [Text-level discourse parsing with rich linguistic features](https://md.research.elamur.ai/free/en/papers/feng-2012-text-level-discourse-parsing-rich/). In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 60–68. Association for Computational Linguistics.
7. Hugo Hernault, Helmut Prendinger, Mitsuru Ishizuka, et al. 2010. [Hilda: a discourse parser using support vector machine classification](https://md.research.elamur.ai/free/en/papers/hernault-2010-hilda-discourse-parser-support-vector/). Dialogue & Discourse, 1(3).
8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
9. Yangfeng Ji and Jacob Eisenstein. 2014. [Representation learning for text-level discourse parsing](https://md.research.elamur.ai/free/en/papers/ji-2014-representation-learning-text-level-discourse/). In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, volume 1, pages 13–24.
10. Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In IJCAI, volume 5, pages 1085–1090. [find](https://scholar.google.com/scholar?q=Mirella+Lapata+and+Regina+Barzilay%2E+2005%2E+Automatic+evaluation+of+text+coherence%3A+Models+and+representations%2E+In+IJCAI%2C+volume+5%2C+pages+1085%E2%80%931090%2E)
11. Alex Lascarides and Nicholas Asher. 1991. [Discourse relations and defeasible knowledge](https://md.research.elamur.ai/free/en/papers/lascarides-1991-discourse-relations-defeasible-knowledge/). In Proceedings of the 29th annual meeting on Association for Computational Linguistics, pages 55–62. Association for Computational Linguistics.
12. Huong LeThanh, Geetha Abeysinghe, and Christian Huyck. 2004. Generating discourse structures for written texts. In Proceedings of the 20th international conference on Computational Linguistics, page 329. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Huong+LeThanh%2C+Geetha+Abeysinghe%2C+and+Christian+Huyck%2E+2004%2E+Generating+discourse+structures+for+written+texts%2E+In+Proceedings+of+the+20th+international+conference+on+Computational+Linguistics%2C+page+3)
13. Jiwei Li and Eduard Hovy. 2014. A model of coherence based on distributed sentence representation. [find](https://scholar.google.com/scholar?q=Jiwei+Li+and+Eduard+Hovy%2E+2014%2E+A+model+of+coherence+based+on+distributed+sentence+representation%2E)
14. Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. [Recursive deep models for discourse parsing](https://md.research.elamur.ai/free/en/papers/li-2014-recursive-deep-models-discourse-parsing/). In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069.
15. Chin-Yew Lin and Eduard Hovy. 2003. [Automatic evaluation of summaries using n-gram co-occurrence statistics](https://md.research.elamur.ai/free/en/papers/lin-2003-automatic-evaluation-summaries-n-gram/). In Proceedings of the 2003
16. Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 71–78. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Conference+of+the+North+American+Chapter+of+the+Association+for+Computational+Linguistics+on+Human+Language+Technology%2DVolume+1%2C+pages+71%E2%80%9378%2E+Association+for+Computational+Linguistics%2E)
17. Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. [Automatically evaluating text coherence using discourse relations](https://md.research.elamur.ai/free/en/papers/lin-2011-automatically-evaluating-text-coherence-discourse/). In Proceedings of the 49th An-2008. nual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 997–1006. Association for Computational Li…
18. Chin-Yew Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81.
19. Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2015. [Addressing the rare word problem in neural machine translation](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/). ACL.
20. William C Mann and Sandra A Thompson. 1988. [Rhetorical structure theory: Toward a functional theory of text organization](https://md.research.elamur.ai/free/en/papers/mann-1988-rhetorical-structure-theory-functional/). Text, 8(3):243–281.
21. Daniel Marcu. 2000. [The rhetorical parsing of unrestricted texts: A surface-based approach](https://md.research.elamur.ai/free/en/papers/marcu-2000-rhetorical-parsing-unrestricted-texts-surface/). Computational linguistics, 26(3):395–448.
22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. [Bleu: a method for automatic evaluation of machine translation](https://md.research.elamur.ai/free/en/papers/papineni-2002-bleu-method-automatic-evaluation-machine/). In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
23. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. [Sequence to sequence learning with neural networks](https://md.research.elamur.ai/free/en/papers/sutskever-2014-sequence-sequence-learning-neural-networks/). In Advances in Neural Information Processing Systems, pages 3104–3112.
24. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2014. Grammar as a foreign language. arXiv preprint arXiv:1412.7449. [find](https://scholar.google.com/scholar?q=Oriol+Vinyals%2C+Lukasz+Kaiser%2C+Terry+Koo%2C+Slav+Petrov%2C+Ilya+Sutskever%2C+and+Geoffrey+Hinton%2E+2014%2E+Grammar+as+a+foreign+language%2E+arXiv+preprint+arXiv%3A1412%2E7449%2E)
25. Florian Wolf and Edward Gibson. 2005. [Representing discourse coherence: A corpus-based study](https://md.research.elamur.ai/free/en/papers/wolf-2005-representing-discourse-coherence-corpus-based/). Computational Linguistics, 31(2):249–287.
26. Kelvin Xu, Jimmy Ba, Ryan Kiros, Aaron Courville, Ruslan Salakhutdinov, Richard Zemel, and Yoshua Bengio. 2015. [Show, attend and tell: Neural image caption generation with visual attention](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/). arXiv preprint arXiv:1502.03044.
27. Rui Yan, Liang Kong, Congrui Huang, Xiaojun Wan, Xiaoming Li, and Yan Zhang. 2011a. Timeline generation through evolutionary trans-temporal summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 433–443. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Rui+Yan%2C+Liang+Kong%2C+Congrui+Huang%2C+Xiaojun+Wan%2C+Xiaoming+Li%2C+and+Yan+Zhang%2E+2011a%2E+Timeline+generation+through+evolutionary+trans%2Dtemporal+summarization%2E+In+Proceedings+of+the+Conference+on+Empirical)
28. Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. 2011b. Evolutionary timeline summarization: a balanced optimization framework via iterative substitution. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieva… [find](https://scholar.google.com/scholar?q=Rui+Yan%2C+Xiaojun+Wan%2C+Jahna+Otterbacher%2C+Liang+Kong%2C+Xiaoming+Li%2C+and+Yan+Zhang%2E+2011b%2E+Evolutionary+timeline+summarization%3A+a+balanced+optimization+framework+via+iterative+substitution%2E+In+Proceeding)

*Updated 2 November 2026*