---
title: "Jointly Learning to Extract and Compress — Berg-Kirkpatrick 2011"
description: "The authors instead train one model that directly targets final summary quality and that both selects… 28 references, cited by 94 papers here."
authors: ["Taylor Berg-Kirkpatrick", "Dan Gillick", "Dan Klein"]
year: 2011
venue: "Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/P11-1049/"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kirkpatrick-2011-jointly-learning-extract-compress/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kirkpatrick-2011-jointly-learning-extract-compress/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2011](https://md.research.elamur.ai/free/en/papers/2011/) / Jointly Learning to Extract and Compress

# Jointly Learning to Extract and Compress

**Authors:** [Taylor Berg-Kirkpatrick](https://md.research.elamur.ai/free/en/authors/taylor-berg-kirkpatrick/), [Dan Gillick](https://md.research.elamur.ai/free/en/authors/dan-gillick/) and [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/) · **Year:** 2011 · **Venue:** [Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies](https://md.research.elamur.ai/free/en/venues/acl/) ([ACL 2011](https://md.research.elamur.ai/free/en/venues/acl/2011/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P11-1049/) · [PDF](https://aclanthology.org/P11-1049.pdf)

**Citations:** [94](https://md.research.elamur.ai/free/en/papers/kirkpatrick-2011-jointly-learning-extract-compress/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles multi-document summarization. Earlier learned systems usually trained intermediate scorers, such as per-sentence or per-word predictors, and then assembled summaries with heuristics. The authors instead train one model that directly targets final summary quality and that both selects and shortens sentences.

A candidate summary gets a linear score built from two kinds of features:
- features of the bigram types it contains;
- features of the subtree deletions made in constituency parses of the chosen sentences.

Weights are learned with a structured soft-margin SVM. Its loss is bigram recall against reference summaries, a ROUGE-like measure. Because the space of possible summaries is exponential, constraints are added incrementally with a cutting-plane procedure. Decoding is an ILP, and a faster approximate decoder performs similarly. For training, the authors created a new manually annotated set of extracted and compressed sentences.

On TAC 2008, the joint system beats both the unlearned extractive baseline and a learned extraction-only variant on ROUGE and Pyramid, with no loss in judged linguistic quality. The authors report the best published ROUGE on this data at the time.

## Key points
- **Three systems compared:**
  - EXTRACTIVE BASELINE: a reimplementation of Gillick and Favre (2009), where each bigram is valued by the number of input documents containing it.
  - LEARNED EXTRACTIVE.
  - LEARNED COMPRESSIVE, the joint model. A LAST DOCUMENT baseline is also included.
- **Compression mechanism:** each parse node has an indicator variable, and a node can be kept only if its parent is kept. Every broken parent–child edge deletes a whole subtree and contributes its own learned feature score.
- **Training choices:** online learners (perceptron, MIRA) proved unstable, so the authors used batch structured SVM training with cutting planes. Since bigram recall factors over bigrams, loss-augmented inference reuses the ordinary decoder.
- **Evaluation:** summaries are limited to 100 words, each built from 10-document TAC 2008 clusters. Metrics were ROUGE, Pyramid, and linguistic quality judged on Mechanical Turk (two judgments per summary, Gillick and Liu 2010 template).
- **Output statistics:**

  | System | Sentences per summary | Words per sentence | Non-stopword types |
  |---|---|---|---|
  | LEARNED COMPRESSIVE | 4.5 | 22.9 | 38.8 |
  | EXTRACTIVE BASELINE | 5.0 | 20.8 | 36.3 |

## Interesting ideas
- **Train on the end metric.** Making the training loss a decomposable stand-in for the final evaluation measure (here bigram recall) removes the mismatch between what is learned and what is evaluated. This could apply to any pipeline task with a structured output.
- **Pick a loss that factors like the model.** When the loss decomposes over the same parts as the model features, loss-augmented decoding comes for free with the existing decoder. That is a practical design rule for structured learning.
- **Handle redundancy through coverage.** Counting each bigram type at most once lets the objective discourage repetition naturally, with no separate redundancy heuristic. This is worth trying for selection tasks such as building balanced corpora or picking diverse examples.
- **Compress by deleting parse subtrees.** Expressing compression as parent-dependent keep/cut variables on a parse tree keeps outputs mostly grammatical and lets deletions get learned scores. The idea could carry over to Ukrainian treebank-based sentence simplification.
- **Use crowd judgments for one dimension only.** Mechanical Turk judgments were used to check linguistic quality, since prior work found them reliable for that and weaker for content. Content was left to ROUGE and Pyramid, which is a cheap way to confirm that gains do not cost readability.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (2): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works

## Cite

Cite as: Taylor Berg-Kirkpatrick, Dan Gillick and Dan Klein. 2011. Jointly Learning to Extract and Compress. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{bergkirkpatrick2011jointly,
  title = {{Jointly Learning to Extract and Compress}},
  author = {Taylor Berg-Kirkpatrick, Dan Gillick and Dan Klein},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-1049/}
}
```

- [← previous Types of Common-Sense Knowledge Needed for Recognizing Textual Entailment](https://md.research.elamur.ai/free/en/papers/lobue-2011-types-common-sense-knowledge-needed/)
- [next → A Discriminative Approach to Japanese Zero Anaphora Resolution with Large-scale Lexicalized Case Frames](https://md.research.elamur.ai/free/en/papers/sasano-2011-discriminative-approach-japanese-zero-anaphora/)
- [↑ back to papers of 2011](https://md.research.elamur.ai/free/en/papers/2011/)

## References

28 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. J. Carbonell and J. Goldstein. 1998. The use of MMR, diversity-based reranking for reordering documents and producing summaries. In Proc. of SIGIR. [find](https://scholar.google.com/scholar?q=J%2E+Carbonell+and+J%2E+Goldstein%2E+1998%2E+The+use+of+MMR%2C+diversity%2Dbased+reranking+for+reordering+documents+and+producing+summaries%2E+In+Proc%2E+of+SIGIR%2E)
2. D. Chiang, Y. Marton, and P. Resnik. 2008. Online large-margin training of syntactic and structural translation features. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=D%2E+Chiang%2C+Y%2E+Marton%2C+and+P%2E+Resnik%2E+2008%2E+Online+large%2Dmargin+training+of+syntactic+and+structural+translation+features%2E+In+Proc%2E+of+EMNLP%2E)
3. J. Clarke and M. Lapata. 2008. Global Inference for Sentence Compression: An Integer Linear Programming Approach. Journal of Artificial Intelligence Research, 31:399–429. [find](https://scholar.google.com/scholar?q=J%2E+Clarke+and+M%2E+Lapata%2E+2008%2E+Global+Inference+for+Sentence+Compression%3A+An+Integer+Linear+Programming+Approach%2E+Journal+of+Artificial+Intelligence+Research%2C+31%3A399%E2%80%93429%2E)
4. K. Crammer and Y. Singer. 2003. Ultraconservative online algorithms for multiclass problems. Journal of Machine Learning Research, 3:951–991. [find](https://scholar.google.com/scholar?q=K%2E+Crammer+and+Y%2E+Singer%2E+2003%2E+Ultraconservative+online+algorithms+for+multiclass+problems%2E+Journal+of+Machine+Learning+Research%2C+3%3A951%E2%80%93991%2E)
5. H.C. Daumé III. 2006. Practical structured learning techniques for natural language processing. Ph.D. thesis, University of Southern California. [find](https://scholar.google.com/scholar?q=H%2EC%2E+Daume%CC%81+III%2E+2006%2E+Practical+structured+learning+techniques+for+natural+language+processing%2E+Ph%2ED%2E+thesis%2C+University+of+Southern+California%2E)
6. D. Gillick and B. Favre. 2009. [A scalable global model for summarization](https://md.research.elamur.ai/free/en/papers/gillick-2009-scalable-global-model-summarization/). In Proc. of ACL Workshop on Integer Linear Programming for Natural Language Processing.
7. D. Gillick and Y. Liu. 2010. [Non-Expert Evaluation of Summarization Systems is Risky](https://md.research.elamur.ai/free/en/papers/burch-2010-creating-speech-language-data-amazon/). In Proc. of NAACL Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk.
8. K. Knight and D. Marcu. 2001. Statistics-based summarization-step one: Sentence compression. In Proc. of AAAI. [find](https://scholar.google.com/scholar?q=K%2E+Knight+and+D%2E+Marcu%2E+2001%2E+Statistics%2Dbased+summarization%2Dstep+one%3A+Sentence+compression%2E+In+Proc%2E+of+AAAI%2E)
9. L. Li, K. Zhou, G.R. Xue, H. Zha, and Y. Yu. 2009. Enhancing diversity, coverage and balance for summarization through structure learning. In Proc. of the 18th International Conference on World Wide Web. [find](https://scholar.google.com/scholar?q=L%2E+Li%2C+K%2E+Zhou%2C+G%2ER%2E+Xue%2C+H%2E+Zha%2C+and+Y%2E+Yu%2E+2009%2E+Enhancing+diversity%2C+coverage+and+balance+for+summarization+through+structure+learning%2E+In+Proc%2E+of+the+18th+International+Conference+on+World+Wide+W)
10. P. Liang, A. Bouchard-Côté, D. Klein, and B. Taskar. 2006. An end-to-end discriminative approach to machine translation. In Proc. of the ACL. [find](https://scholar.google.com/scholar?q=P%2E+Liang%2C+A%2E+Bouchard%2DCo%CC%82te%CC%81%2C+D%2E+Klein%2C+and+B%2E+Taskar%2E+2006%2E+An+end%2Dto%2Dend+discriminative+approach+to+machine+translation%2E+In+Proc%2E+of+the+ACL%2E)
11. C.Y. Lin. 2003. Improving summarization performance by sentence compression: a pilot study. In Proc. of ACL Workshop on Information Retrieval with Asian Languages. [find](https://scholar.google.com/scholar?q=C%2EY%2E+Lin%2E+2003%2E+Improving+summarization+performance+by+sentence+compression%3A+a+pilot+study%2E+In+Proc%2E+of+ACL+Workshop+on+Information+Retrieval+with+Asian+Languages%2E)
12. C.Y. Lin. 2004. [Rouge: A package for automatic evaluation of summaries](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/). In Proc. of ACL Workshop on Text Summarization Branches Out.
13. A.F.T. Martins and N.A. Smith. 2009. Summarization with a joint model for sentence extraction and compression. In Proc. of NAACL Workshop on Integer Linear Programming for Natural Language Processing. [find](https://scholar.google.com/scholar?q=A%2EF%2ET%2E+Martins+and+N%2EA%2E+Smith%2E+2009%2E+Summarization+with+a+joint+model+for+sentence+extraction+and+compression%2E+In+Proc%2E+of+NAACL+Workshop+on+Integer+Linear+Programming+for+Natural+Language+Processing%2E)
14. R. McDonald. 2006. Discriminative sentence compression with soft syntactic constraints. In Proc. of EACL. [find](https://scholar.google.com/scholar?q=R%2E+McDonald%2E+2006%2E+Discriminative+sentence+compression+with+soft+syntactic+constraints%2E+In+Proc%2E+of+EACL%2E)
15. A. Nenkova and R. Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=A%2E+Nenkova+and+R%2E+Passonneau%2E+2004%2E+Evaluating+content+selection+in+summarization%3A+The+pyramid+method%2E+In+Proc%2E+of+NAACL%2E)
16. A. Nenkova and L. Vanderwende. 2005. The impact of frequency on summarization. Technical report, MSR-TR-2005-101. Redmond, Washington: Microsoft Research. [find](https://scholar.google.com/scholar?q=A%2E+Nenkova+and+L%2E+Vanderwende%2E+2005%2E+The+impact+of+frequency+on+summarization%2E+Technical+report%2C+MSR%2DTR%2D2005%2D101%2E+Redmond%2C+Washington%3A+Microsoft+Research%2E)
17. S. Petrov and D. Klein. 2007. Learning and inference for hierarchically split PCFGs. In AAAI. [find](https://scholar.google.com/scholar?q=S%2E+Petrov+and+D%2E+Klein%2E+2007%2E+Learning+and+inference+for+hierarchically+split+PCFGs%2E+In+AAAI%2E)
18. J.C. Platt. 1999. Fast training of support vector machines using sequential minimal optimization. In Advances in Kernel Methods. MIT press. [find](https://scholar.google.com/scholar?q=J%2EC%2E+Platt%2E+1999%2E+Fast+training+of+support+vector+machines+using+sequential+minimal+optimization%2E+In+Advances+in+Kernel+Methods%2E+MIT+press%2E)
19. F. Schilder and R. Kondadadi. 2008. Fastsum: Fast and accurate query-based multi-document summarization. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=F%2E+Schilder+and+R%2E+Kondadadi%2E+2008%2E+Fastsum%3A+Fast+and+accurate+query%2Dbased+multi%2Ddocument+summarization%2E+In+Proc%2E+of+ACL%2E)
20. D. Shen, J.T. Sun, H. Li, Q. Yang, and Z. Chen. 2007. Document summarization using conditional random fields. In Proc. of IJCAI. [find](https://scholar.google.com/scholar?q=D%2E+Shen%2C+J%2ET%2E+Sun%2C+H%2E+Li%2C+Q%2E+Yang%2C+and+Z%2E+Chen%2E+2007%2E+Document+summarization+using+conditional+random+fields%2E+In+Proc%2E+of+IJCAI%2E)
21. B. Taskar, C. Guestrin, and D. Koller. 2003. Max-margin Markov networks. In Proc. of NIPS. [find](https://scholar.google.com/scholar?q=B%2E+Taskar%2C+C%2E+Guestrin%2C+and+D%2E+Koller%2E+2003%2E+Max%2Dmargin+Markov+networks%2E+In+Proc%2E+of+NIPS%2E)
22. B. Taskar, D. Klein, M. Collins, D. Koller, and C. Manning. 2004. Max-margin parsing. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=B%2E+Taskar%2C+D%2E+Klein%2C+M%2E+Collins%2C+D%2E+Koller%2C+and+C%2E+Manning%2E+2004%2E+Max%2Dmargin+parsing%2E+In+Proc%2E+of+EMNLP%2E)
23. S. Teufel and M. Moens. 1997. Sentence extraction as a classification task. In Proc. of ACL Workshop on Intelligent and Scalable Text Summarization. [find](https://scholar.google.com/scholar?q=S%2E+Teufel+and+M%2E+Moens%2E+1997%2E+Sentence+extraction+as+a+classification+task%2E+In+Proc%2E+of+ACL+Workshop+on+Intelligent+and+Scalable+Text+Summarization%2E)
24. I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun. 2004. Support vector machine learning for interdependent and structured output spaces. In Proc. of ICML. [find](https://scholar.google.com/scholar?q=I%2E+Tsochantaridis%2C+T%2E+Hofmann%2C+T%2E+Joachims%2C+and+Y%2E+Altun%2E+2004%2E+Support+vector+machine+learning+for+interdependent+and+structured+output+spaces%2E+In+Proc%2E+of+ICML%2E)
25. V.N. Vapnik. 1998. Statistical learning theory. John Wiley and Sons, New York. [find](https://scholar.google.com/scholar?q=V%2EN%2E+Vapnik%2E+1998%2E+Statistical+learning+theory%2E+John+Wiley+and+Sons%2C+New+York%2E)
26. K. Woodsend and M. Lapata. 2010. Automatic generation of story highlights. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=K%2E+Woodsend+and+M%2E+Lapata%2E+2010%2E+Automatic+generation+of+story+highlights%2E+In+Proc%2E+of+ACL%2E)
27. W. Yih, J. Goodman, L. Vanderwende, and H. Suzuki. 2007. Multi-document summarization by maximizing informative content-words. In Proc. of IJCAI. [find](https://scholar.google.com/scholar?q=W%2E+Yih%2C+J%2E+Goodman%2C+L%2E+Vanderwende%2C+and+H%2E+Suzuki%2E+2007%2E+Multi%2Ddocument+summarization+by+maximizing+informative+content%2Dwords%2E+In+Proc%2E+of+IJCAI%2E)
28. D.M. Zajic, B.J. Dorr, R. Schwartz, and J. Lin. 2006. Sentence compression as a component of a multi-document summarization system. In Proc. of the 2006 Document Understanding Workshop. [find](https://scholar.google.com/scholar?q=D%2EM%2E+Zajic%2C+B%2EJ%2E+Dorr%2C+R%2E+Schwartz%2C+and+J%2E+Lin%2E+2006%2E+Sentence+compression+as+a+component+of+a+multi%2Ddocument+summarization+system%2E+In+Proc%2E+of+the+2006+Document+Understanding+Workshop%2E)

*Updated 24 October 2026*