---
title: "Sentence Compression by Deletion with LSTMs — Filippova 2015"
description: "Most deletion-based sentence compressors depend on parsers, PoS taggers, or other linguistic preprocessing."
authors: ["Katja Filippova", "Enrique Alfonseca", "Carlos A. Colmenares", "Lukasz Kaiser", "Oriol Vinyals"]
year: 2015
venue: "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D15-1042/"
doi: "10.18653/v1/d15-1042"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/filippova-2015-sentence-compression-deletion-lstms/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/filippova-2015-sentence-compression-deletion-lstms/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / Sentence Compression by Deletion with LSTMs

# Sentence Compression by Deletion with LSTMs

**Authors:** [Katja Filippova](https://md.research.elamur.ai/free/en/authors/katja-filippova/), [Enrique Alfonseca](https://md.research.elamur.ai/free/en/authors/enrique-alfonseca/), Carlos A. Colmenares, [Lukasz Kaiser](https://md.research.elamur.ai/free/en/authors/lukasz-kaiser/), [Oriol Vinyals](https://md.research.elamur.ai/free/en/authors/oriol-vinyals/) · **Year:** 2015 · **Venue:** Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D15-1042/) · [PDF](https://aclanthology.org/D15-1042.pdf) · [DOI 10.18653/v1/d15-1042](https://doi.org/10.18653/v1/d15-1042) · [OpenAlex](https://openalex.org/W2251654079)

**Citations:** 241 citations (openalex-snapshot · 2026-10-14) · [96](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Most deletion-based sentence compressors depend on parsers, PoS taggers, or other linguistic preprocessing. Parsing errors then flow into the output: state-of-the-art parsers reach only about 91 points of labeled attachment accuracy. Filippova and colleagues at Google Research ask whether a strong compressor can work from tokens alone. They treat compression as sequence-to-sequence labeling. A three-layer stacked LSTM with dropout reads the sentence in reverse, then emits a keep (1), drop (0), or EOS decision for each token. The only input is pre-trained word2vec embeddings, with no tags, dependencies, or target length. The model is trained on fewer than two million sentence–compression pairs harvested automatically from the web. It is compared with a large-scale reimplementation of McDonald (2006) that uses rich syntactic features. In human evaluation, the plain LSTM beats this feature-heavy baseline on both readability and informativeness. The result suggests that compression research can depend much less on explicit syntactic modeling.

## Key points
- The basic token-only LSTM exactly reproduces about 30% of gold compressions on a large test set (accuracy@1).
- Human raters scored the basic LSTM 4.5/5 for readability and 3.8/5 for informativeness on 200 evaluated sentences, outperforming the baseline.
- The baseline is McDonald's (2006) model, rebuilt for large data in three ways:
  - Its training uses a structured perceptron spread across multiple machines, with parameters repeatedly averaged across them, and MIRA with K=1.
  - It uses 463,614 dependency-based features.
  - It includes optional 256-dimensional Skipgram embeddings.
- An extended variant that adds syntactic information (LSTM+PAR+PRES) did worse with human judges than the basic model. The authors think it overfit to the tree-pruning procedure used to build the gold data.
- The main remaining errors involve:
  - quotes, intervening commas, and unusual punctuation;
  - sentences with little removable material;
  - cases where context is essential to interpret the event.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

96 papers of our library cite it (of the 140884 whose bibliographies we read); 12 of them have a page here, 84 not yet.

- [Globally Normalized Transition-Based Neural Networks](https://md.research.elamur.ai/free/en/papers/andor-2016-globally-normalized-transition-based-neural/)
- [Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting](https://md.research.elamur.ai/free/en/papers/chen-2018-fast-abstractive-summarization-reinforce-selected/)
- [Abstractive Sentence Summarization with Attentive Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/chopra-2016-abstractive-sentence-summarization-attentive-recurrent/)
- [Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategies](https://md.research.elamur.ai/free/en/papers/grusky-2018-newsroom-dataset-1-3-million/)
- [Controlling Output Length in Neural Encoder-Decoders](https://md.research.elamur.ai/free/en/papers/kikuchi-2016-controlling-output-length-neural-encoder/)
- [Controllable Text Simplification with Explicit Paraphrasing](https://md.research.elamur.ai/free/en/papers/maddela-2021-controllable-text-simplification-explicit-paraphrasing/)
- [Encode, Tag, Realize: High-Precision Text Editing](https://md.research.elamur.ai/free/en/papers/malmi-2019-encode-tag-realize-high-precision/)
- [Ranking Sentences for Extractive Summarization with Reinforcement Learning](https://md.research.elamur.ai/free/en/papers/narayan-2018-ranking-sentences-extractive-summarization-reinforcement/)
- [Multi-Reward Reinforced Summarization with Saliency and Entailment](https://md.research.elamur.ai/free/en/papers/pasunuru-2018-multi-reward-reinforced-summarization-saliency/)
- [Optimizing Statistical Machine Translation for Text Simplification](https://md.research.elamur.ai/free/en/papers/xu-2016-optimizing-statistical-machine-translation-text/)
- [From Shakespeare to Twitter: What are Language Styles all about?](https://md.research.elamur.ai/free/en/papers/xu-2017-shakespeare-twitter-what-are-language/)
- [Idiomatic Expression Identification using Semantic Compatibility](https://md.research.elamur.ai/free/en/papers/zeng-2021-idiomatic-expression-identification-semantic-compatibility/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{filippova2015sentence,
  title = {{Sentence Compression by Deletion with LSTMs}},
  author = {Filippova, Katja and Alfonseca, Enrique and Colmenares, Carlos A. and Kaiser, Lukasz and Vinyals, Oriol},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1042},
  url = {https://doi.org/10.18653/v1/d15-1042}
}
```

- [← previous SensEmbed: Learning Sense Embeddings for Word and Relational Similarity](https://md.research.elamur.ai/free/en/papers/iacobacci-2015-sensembed-learning-sense-embeddings-word/)
- [next → Sequence Level Training with Recurrent Neural Networks](https://md.research.elamur.ai/free/en/papers/ranzato-2015-sequence-level-training-recurrent-neural/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

25 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Berg-Kirkpatrick, T., D. Gillick & D. Klein (2011). Jointly learning to extract and compress. In Proc. of ACL-11. [find](https://scholar.google.com/scholar?q=Berg%2DKirkpatrick%2C+T%2E%2C+D%2E+Gillick+%26+D%2E+Klein+%282011%29%2E+Jointly+learning+to+extract+and+compress%2E+In+Proc%2E+of+ACL%2D11%2E)
2. Cho, K., B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk & Y. Bengio (2014). Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25… [find](https://scholar.google.com/scholar?q=Cho%2C+K%2E%2C+B%2E+van+Merrienboer%2C+C%2E+Gulcehre%2C+D%2E+Bahdanau%2C+F%2E+Bougares%2C+H%2E+Schwenk+%26+Y%2E+Bengio+%282014%29%2E+Learning+phrase+representations+using+rnn+encoder%2Ddecoder+for+statistical+machine+translation%2E+In+Pro)
3. Clarke, J. & M. Lapata (2006). Models for sentence compression: A comparison across domains, training requirements and evaluation measures. In Proc. of COLING-ACL-06, pp. 377–385. [find](https://scholar.google.com/scholar?q=Clarke%2C+J%2E+%26+M%2E+Lapata+%282006%29%2E+Models+for+sentence+compression%3A+A+comparison+across+domains%2C+training+requirements+and+evaluation+measures%2E+In+Proc%2E+of+COLING%2DACL%2D06%2C+pp%2E+377%E2%80%93385%2E)
4. Clarke, J. & M. Lapata (2008). Global inference for sentence compression: An integer linear programming approach. Journal of Artificial Intelligence Research, 31:399–429. [find](https://scholar.google.com/scholar?q=Clarke%2C+J%2E+%26+M%2E+Lapata+%282008%29%2E+Global+inference+for+sentence+compression%3A+An+integer+linear+programming+approach%2E+Journal+of+Artificial+Intelligence+Research%2C+31%3A399%E2%80%93429%2E)
5. Cohn, T. & M. Lapata (2008). Sentence compression beyond word deletion. In Proc. of COLING-08, pp. 137–144. [find](https://scholar.google.com/scholar?q=Cohn%2C+T%2E+%26+M%2E+Lapata+%282008%29%2E+Sentence+compression+beyond+word+deletion%2E+In+Proc%2E+of+COLING%2D08%2C+pp%2E+137%E2%80%93144%2E)
6. Crammer, K. & Y. Singer (2003). Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951– 991. [find](https://scholar.google.com/scholar?q=Crammer%2C+K%2E+%26+Y%2E+Singer+%282003%29%2E+Ultraconservative+online+algorithms+for+multiclass+problems%2E+Journal+of+Machine+Learning+Research%2C+3%3A951%E2%80%93+991%2E)
7. Filippova, K. & Y. Altun (2013). Overcoming the lack of parallel data in sentence compression. In Proc. of EMNLP-13, pp. 1481–1491. [find](https://scholar.google.com/scholar?q=Filippova%2C+K%2E+%26+Y%2E+Altun+%282013%29%2E+Overcoming+the+lack+of+parallel+data+in+sentence+compression%2E+In+Proc%2E+of+EMNLP%2D13%2C+pp%2E+1481%E2%80%931491%2E)
8. Hochreiter, S. & J. Schmidhuber (1997). Long short-term memory. Neural computation, 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Hochreiter%2C+S%2E+%26+J%2E+Schmidhuber+%281997%29%2E+Long+short%2Dterm+memory%2E+Neural+computation%2C+9%288%29%3A1735%E2%80%931780%2E)
9. Jing, H. (2000). Sentence reduction for automatic text summarization. In Proc. of ANLP-00, pp. 310–315. [find](https://scholar.google.com/scholar?q=Jing%2C+H%2E+%282000%29%2E+Sentence+reduction+for+automatic+text+summarization%2E+In+Proc%2E+of+ANLP%2D00%2C+pp%2E+310%E2%80%93315%2E)
10. Knight, K. & D. Marcu (2000). Statistics-based summarization – step one: Sentence compression. In Proc. of AAAI-00, pp. 703–711. [find](https://scholar.google.com/scholar?q=Knight%2C+K%2E+%26+D%2E+Marcu+%282000%29%2E+Statistics%2Dbased+summarization+%E2%80%93+step+one%3A+Sentence+compression%2E+In+Proc%2E+of+AAAI%2D00%2C+pp%2E+703%E2%80%93711%2E)
11. Li, J. & D. Jurafsky (2015). A hierarchical LSTM autoencoder for paragraphs and documents. In Proc. of ACL-15. [find](https://scholar.google.com/scholar?q=Li%2C+J%2E+%26+D%2E+Jurafsky+%282015%29%2E+A+hierarchical+LSTM+autoencoder+for+paragraphs+and+documents%2E+In+Proc%2E+of+ACL%2D15%2E)
12. Luong, M.-T., I. Sutskever, Q. V. Le, O. Vinyals & W. Zaremba (2014). Addressing the rare word problem in neural machine translation. CoRR, abs/1410.8206. [find](https://scholar.google.com/scholar?q=Luong%2C+M%2E%2DT%2E%2C+I%2E+Sutskever%2C+Q%2E+V%2E+Le%2C+O%2E+Vinyals+%26+W%2E+Zaremba+%282014%29%2E+Addressing+the+rare+word+problem+in+neural+machine+translation%2E+CoRR%2C+abs%2F1410%2E8206%2E) [our card](https://md.research.elamur.ai/free/en/papers/luong-2015-addressing-rare-word-problem-neural/)
13. McDonald, R. (2006). Discriminative sentence compression with soft syntactic evidence. In Proc. of EACL-06, pp. 297–304. [find](https://scholar.google.com/scholar?q=McDonald%2C+R%2E+%282006%29%2E+Discriminative+sentence+compression+with+soft+syntactic+evidence%2E+In+Proc%2E+of+EACL%2D06%2C+pp%2E+297%E2%80%93304%2E)
14. McDonald, R., K. Crammer & F. Pereira (2005). Online large-margin training of dependency parsers. In Proc. of ACL-05, pp. 91–98. [find](https://scholar.google.com/scholar?q=McDonald%2C+R%2E%2C+K%2E+Crammer+%26+F%2E+Pereira+%282005%29%2E+Online+large%2Dmargin+training+of+dependency+parsers%2E+In+Proc%2E+of+ACL%2D05%2C+pp%2E+91%E2%80%9398%2E) [our card](https://md.research.elamur.ai/free/en/papers/narrative-2005-mcdonald-online-large-margin-training-dependency/)
15. McDonald, R., K. Hall & G. Mann (2010). Distributed training strategies for the structured perceptron. In Proc. of NAACL-HLT-10, pp. 456–464. [find](https://scholar.google.com/scholar?q=McDonald%2C+R%2E%2C+K%2E+Hall+%26+G%2E+Mann+%282010%29%2E+Distributed+training+strategies+for+the+structured+perceptron%2E+In+Proc%2E+of+NAACL%2DHLT%2D10%2C+pp%2E+456%E2%80%93464%2E)
16. Mikolov, T., I. Sutskever, K. Chen, G. S. Corrado & J. Dean (2013). Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pp. 3111–3119. [find](https://scholar.google.com/scholar?q=Mikolov%2C+T%2E%2C+I%2E+Sutskever%2C+K%2E+Chen%2C+G%2E+S%2E+Corrado+%26+J%2E+Dean+%282013%29%2E+Distributed+representations+of+words+and+phrases+and+their+compositionality%2E+In+Advances+in+Neural+Information+Processing+Systems%2C+p)
17. Napoles, C., C. Callison-Burch & B. Van Durme (2011). Evaluating sentence compression: Pitfalls and suggested remedies. In Proceedings of the Workshop on Monolingual Text-to-text Generation, Portland, OR, June 24 2011, pp. 91–97. [find](https://scholar.google.com/scholar?q=Napoles%2C+C%2E%2C+C%2E+Callison%2DBurch+%26+B%2E+Van+Durme+%282011%29%2E+Evaluating+sentence+compression%3A+Pitfalls+and+suggested+remedies%2E+In+Proceedings+of+the+Workshop+on+Monolingual+Text%2Dto%2Dtext+Generation%2C+Portland%2C)
18. Riezler, S., T. H. King, R. Crouch & A. Zaenen (2003). Statistical sentence condensation using ambiguity packing and stochastic disambiguation methods for Lexical-Functional Grammar. In Proc. of HLT-NAACL-03, pp. 118–125. [find](https://scholar.google.com/scholar?q=Riezler%2C+S%2E%2C+T%2E+H%2E+King%2C+R%2E+Crouch+%26+A%2E+Zaenen+%282003%29%2E+Statistical+sentence+condensation+using+ambiguity+packing+and+stochastic+disambiguation+methods+for+Lexical%2DFunctional+Grammar%2E+In+Proc%2E+of+HLT%2DN)
19. Srivastava, N., G. Hinton, A. Krizhevsky, I. Sutskever & R. Salakhutdinov (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. [find](https://scholar.google.com/scholar?q=Srivastava%2C+N%2E%2C+G%2E+Hinton%2C+A%2E+Krizhevsky%2C+I%2E+Sutskever+%26+R%2E+Salakhutdinov+%282014%29%2E+Dropout%3A+A+simple+way+to+prevent+neural+networks+from+overfitting%2E+Journal+of+Machine+Learning+Research%2C+15%281%29%3A1929%E2%80%9319)
20. Sutskever, I., O. Vinyals & Q. V. Le (2014). Sequence to sequence learning with neural networks. In Proc. of NIPS-2014. [find](https://scholar.google.com/scholar?q=Sutskever%2C+I%2E%2C+O%2E+Vinyals+%26+Q%2E+V%2E+Le+%282014%29%2E+Sequence+to+sequence+learning+with+neural+networks%2E+In+Proc%2E+of+NIPS%2D2014%2E)
21. Vinyals, O., A. Toshev, S. Bengio & D. Erhan (2015). Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR-2015. [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E%2C+A%2E+Toshev%2C+S%2E+Bengio+%26+D%2E+Erhan+%282015%29%2E+Show+and+tell%3A+A+neural+image+caption+generator%2E+In+Proceedings+of+the+IEEE+Conference+on+Computer+Vision+and+Pattern+Recognition%2C+CVPR%2D2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/vinyals-2014-show-tell-neural-image-caption/)
22. Vinyals, O., L. Kaiser, T. Koo, S. Petrov, I. Sutskever & G. E. Hinton (2014). Grammar as a foreign language. CoRR, abs/1412.7449. [find](https://scholar.google.com/scholar?q=Vinyals%2C+O%2E%2C+L%2E+Kaiser%2C+T%2E+Koo%2C+S%2E+Petrov%2C+I%2E+Sutskever+%26+G%2E+E%2E+Hinton+%282014%29%2E+Grammar+as+a+foreign+language%2E+CoRR%2C+abs%2F1412%2E7449%2E)
23. Xu, K., J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. Zemel & Y. Bengio (2015). Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of ICML-2015. [find](https://scholar.google.com/scholar?q=Xu%2C+K%2E%2C+J%2E+Ba%2C+R%2E+Kiros%2C+K%2E+Cho%2C+A%2E+Courville%2C+R%2E+Salakhutdinov%2C+R%2E+Zemel+%26+Y%2E+Bengio+%282015%29%2E+Show%2C+attend+and+tell%3A+Neural+image+caption+generation+with+visual+attention%2E+In+Proceedings+of+ICML%2D2015%2E) [our card](https://md.research.elamur.ai/free/en/papers/xu-2015-show-attend-tell-neural-image/)
24. Zaremba, W. & I. Sutskever (2014). Learning to execute. CoRR, abs/1410.4615. [find](https://scholar.google.com/scholar?q=Zaremba%2C+W%2E+%26+I%2E+Sutskever+%282014%29%2E+Learning+to+execute%2E+CoRR%2C+abs%2F1410%2E4615%2E)
25. Zhang, H. & R. McDonald (2014). Enforcing structural diversity in cube-pruned dependency parsing. In Proc. of ACL-14, pp. 656–661. [find](https://scholar.google.com/scholar?q=Zhang%2C+H%2E+%26+R%2E+McDonald+%282014%29%2E+Enforcing+structural+diversity+in+cube%2Dpruned+dependency+parsing%2E+In+Proc%2E+of+ACL%2D14%2C+pp%2E+656%E2%80%93661%2E)