---
title: "BillSum: A Corpus for Automatic — Kornilova 2019"
description: "Legislative text is long, deeply nested and technical."
authors: ["Anastassia Kornilova", "Vladimir Eidelman"]
year: 2019
venue: "Proceedings of the 2nd Workshop on New Frontiers in Summarization"
source: "https://aclanthology.org/D19-5406/"
doi: "10.18653/v1/d19-5406"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/kornilova-2019-billsum-corpus-automatic-summarization-us/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/kornilova-2019-billsum-corpus-automatic-summarization-us/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/) / BillSum: A Corpus for Automatic Summarization of US Legislation

# BillSum: A Corpus for Automatic Summarization of US Legislation

**Authors:** Anastassia Kornilova, [Vladimir Eidelman](https://md.research.elamur.ai/free/en/authors/vladimir-eidelman/) · **Year:** 2019 · **Venue:** Proceedings of the 2nd Workshop on New Frontiers in Summarization · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D19-5406/) · [PDF](https://aclanthology.org/D19-5406.pdf) · [DOI 10.18653/v1/d19-5406](https://doi.org/10.18653/v1/d19-5406) · [OpenAlex](https://openalex.org/W2978041899)

**Citations:** 80 citations (OpenAlex, 2026-10-14) · [93](https://md.research.elamur.ai/free/en/papers/kornilova-2019-billsum-corpus-automatic-summarization-us/cited-by/) in our library, of the 140884 papers whose bibliographies we read

## Summary
Legislative text is long, deeply nested and technical. Every US Congressional bill receives a human summary from the Congressional Research Service (CRS), but most state and local legislatures provide nothing comparable. The authors present BillSum, the first dataset built specifically for summarizing legislation. It pairs mid-length US Congressional bills with their CRS summaries and adds a separate California test set, whose summaries come from the state's Legislative Counsel.

They benchmark extractive approaches. Each sentence is labeled by its overlap with the reference summary and then scored by two classifiers: a random forest over positional and TF-IDF features (DOC) and a domain-adapted BERT classifier (SUM). Summaries are assembled with Maximal Marginal Relevance under a 2,000-character budget.

All supervised models beat unsupervised baselines such as TextRank, LSA and SumBasic, and the BERT-based model does best on Congressional bills. Every system still trails an oracle by a wide margin. Models trained only on federal bills also outperform the baselines on California bills, which suggests such summarizers could serve legislatures that have no reference summaries.

## Key points
- **Corpus size and splits:** BillSum contains 22,218 US bills from the 103rd–115th Congresses (1993–2018), divided into 18,949 train and 3,269 test documents. A further 1,237 California bills from the 2015–2016 session serve as a transfer test set.
- **Filtering:**
  - Only bills of 5,000–20,000 characters were kept, which is roughly 20% of US bills from that period.
  - Summaries are capped at 2,000 characters, a length that covers about 90% of reference summaries.
  - Near-duplicate bills were removed using a 96% cosine-similarity threshold.
- **Sentence labels:** A sentence counts as positive when its ROUGE-2 precision against the reference exceeds 0.1.
- **Models:**
  - DOC is a random forest with 50 estimators.
  - SUM is Bert-Large Uncased, first further pretrained on next-sentence prediction over US training bills (20,000 steps, batch size 32), then fine-tuned for 3 epochs.
  - DOC+SUM averages the two models' probabilities.
  - Final selection uses MMR with weights of 0.7 for relevance and 0.3 for redundancy.
- **Results (ROUGE-2 F):**
  - US bills: SUM 23.88, DOC+SUM 23.83, DOC 21.38, TextRank 17.77, oracle 28.74.
  - California bills: DOC+SUM 21.14, TextRank 18.10, oracle 32.83.

## Interesting ideas
- **Merging short bullets into sentences:** Bullets under 10 words are attached to the preceding sentence. This cheap rule turns heavily structured, bullet-based documents into workable sentence units. It could help with other formatted texts such as regulations, contracts or official Ukrainian legal acts.
- **Leak-proof deduplication:** Near-duplicates are dropped by cosine similarity, and any remaining bills that share a title are all forced into the training split. This keeps reintroduced documents from leaking into evaluation, which is useful for any corpus where the same text is republished repeatedly.
- **Measuring length in characters:** Length is counted in characters rather than words, since word counts are unreliable for messy structured text. Applying the same unit to both inputs and summary budgets keeps the setup consistent.
- **KL-divergence word lists:** The authors compare how likely each word is in summaries versus full texts. The resulting lists show that summaries favour action verbs and agency names, while full texts favour edit and background vocabulary. This is a simple, interpretable diagnostic that complements a black-box neural model and could guide feature design in new domains.
- **Testing transfer to an unannotated target:** A held-out test set from a related but different jurisdiction checks whether a model trained where labels exist works where they do not. This evaluation design fits any low-resource setting, such as adapting from a well-annotated language or region to a sparse one.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) · [Law](https://md.research.elamur.ai/free/en/shelves/law/)

Topics: [Summarization](https://md.research.elamur.ai/free/en/topics/summarization/) · [Legal AI](https://md.research.elamur.ai/free/en/topics/legal-ai/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{kornilova2019billsum,
  title = {{BillSum: A Corpus for Automatic Summarization of US Legislation}},
  author = {Kornilova, Anastassia and Eidelman, Vladimir},
  booktitle = {Proceedings of the 2nd Workshop on New Frontiers in Summarization},
  year = {2019},
  doi = {10.18653/v1/d19-5406},
  url = {https://doi.org/10.18653/v1/d19-5406}
}
```

- [← previous Investigating Multilingual NMT Representations at Scale](https://md.research.elamur.ai/free/en/papers/kudugunta-2019-investigating-multilingual-nmt-representations-scale/)
- [next → Controllable Paraphrase Generation with a Syntactic Exemplar](https://md.research.elamur.ai/free/en/papers/chen-2019-controllable-paraphrase-generation-syntactic-exemplar/)
- [↑ back to papers of 2019](https://md.research.elamur.ai/free/en/papers/2019/page/3/)

## References

30 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Leo Breiman. 2001. Random forests. Mach. Learn., 45(1):5–32. [find](https://scholar.google.com/scholar?q=Leo+Breiman%2E+2001%2E+Random+forests%2E+Mach%2E+Learn%2E%2C+45%281%29%3A5%E2%80%9332%2E)
2. Ziqiang Cao, Furu Wei, Sujian Li, Wenjie Li, Ming Zhou, and WANG Houfeng. 2015. Learning summary prior representation for extractive summarization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 829–833. [find](https://scholar.google.com/scholar?q=Ziqiang+Cao%2C+Furu+Wei%2C+Sujian+Li%2C+Wenjie+Li%2C+Ming+Zhou%2C+and+WANG+Houfeng%2E+2015%2E+Learning+summary+prior+representation+for+extractive+summarization%2E+In+Proceedings+of+the+53rd+Annual+Meeting+of+the+Ass)
3. Jaime Carbonell and Jade Goldstein. 1998. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’98, pages 335–336, New York, NY, USA. ACM. [find](https://scholar.google.com/scholar?q=Jaime+Carbonell+and+Jade+Goldstein%2E+1998%2E+The+use+of+mmr%2C+diversity%2Dbased+reranking+for+reordering+documents+and+producing+summaries%2E+In+Proceedings+of+the+21st+Annual+International+ACM+SIGIR+Conferen)
4. Sumit Chopra, Michael Auli, and Alexander M. Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sumit+Chopra%2C+Michael+Auli%2C+and+Alexander+M%2E+Rush%2E+2016%2E+Abstractive+sentence+summarization+with+attentive+recurrent+neural+networks%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapte) [our card](https://md.research.elamur.ai/free/en/papers/chopra-2016-abstractive-sentence-summarization-attentive-recurrent/)
5. Ed Collins, Isabelle Augenstein, and Sebastian Riedel. 2017. A supervised approach to extractive summarisation of scientific papers. arXiv preprint arXiv:1706.03946. [↗](https://arxiv.org/abs/1706.03946)
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. [find](https://scholar.google.com/scholar?q=Jacob+Devlin%2C+Ming%2DWei+Chang%2C+Kenton+Lee%2C+and+Kristina+Toutanova%2E+2018%2E+Bert%3A+Pre%2Dtraining+of+deep+bidirectional+transformers+for+language+understanding%2E) [our card](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/)
7. Vlad Eidelman, Anastassia Kornilova, and Daniel Argyle. 2018. How predictable is your state? leveraging lexical and contextual information for predicting legislative floor action at the state level. arXiv preprint arXiv:1806.05284. [↗](https://arxiv.org/abs/1806.05284)
8. Sean Gerrish and David M Blei. 2011. Predicting legislative roll calls from text. In Proceedings of the 28th international conference on machine learning (icml-11), pages 489–496. [find](https://scholar.google.com/scholar?q=Sean+Gerrish+and+David+M+Blei%2E+2011%2E+Predicting+legislative+roll+calls+from+text%2E+In+Proceedings+of+the+28th+international+conference+on+machine+learning+%28icml%2D11%29%2C+pages+489%E2%80%93496%2E)
9. Daniel Gillick, Benoit Favre, and Dilek Hakkani-Tür. 2008. The icsi summarization system at tac 2008. In TAC. [find](https://scholar.google.com/scholar?q=Daniel+Gillick%2C+Benoit+Favre%2C+and+Dilek+Hakkani%2DTu%CC%88r%2E+2008%2E+The+icsi+summarization+system+at+tac+2008%2E+In+TAC%2E)
10. Jade Goldstein, Vibhu Mittal, Jaime Carbonell, and Mark Kantrowitz. 2000. Multi-document summarization by sentence extraction. In Proceedings of the 2000 NAACL-ANLPWorkshop on Automatic Summarization - Volume 4, NAACL-ANLP-AutoSum ’00, pages 40–48, Stroudsburg, PA, USA. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Jade+Goldstein%2C+Vibhu+Mittal%2C+Jaime+Carbonell%2C+and+Mark+Kantrowitz%2E+2000%2E+Multi%2Ddocument+summarization+by+sentence+extraction%2E+In+Proceedings+of+the+2000+NAACL%2DANLPWorkshop+on+Automatic+Summarization+)
11. Yihong Gong and Xin Liu. 2001. Generic text summarization using relevance measure and latent semantic analysis. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 19–25. ACM. [find](https://scholar.google.com/scholar?q=Yihong+Gong+and+Xin+Liu%2E+2001%2E+Generic+text+summarization+using+relevance+measure+and+latent+semantic+analysis%2E+In+Proceedings+of+the+24th+annual+international+ACM+SIGIR+conference+on+Research+and+dev)
12. Claire Grover, Ben Hachey, and Ian Hughson. 2004. The HOLJ corpus. supporting summarisation of legal texts. In COLING 2004 5th International Workshop on Linguistically Interpreted Corpora, pages 47–54, Geneva, Switzerland. COLING. [find](https://scholar.google.com/scholar?q=Claire+Grover%2C+Ben+Hachey%2C+and+Ian+Hughson%2E+2004%2E+The+HOLJ+corpus%2E+supporting+summarisation+of+legal+texts%2E+In+COLING+2004+5th+International+Workshop+on+Linguistically+Interpreted+Corpora%2C+pages+47%E2%80%9354)
13. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693– 1701. [find](https://scholar.google.com/scholar?q=Karl+Moritz+Hermann%2C+Tomas+Kocisky%2C+Edward+Grefenstette%2C+Lasse+Espeholt%2C+Will+Kay%2C+Mustafa+Suleyman%2C+and+Phil+Blunsom%2E+2015%2E+Teaching+machines+to+read+and+comprehend%2E+In+Advances+in+Neural+Information) [our card](https://md.research.elamur.ai/free/en/papers/hermann-2015-teaching-machines-read-comprehend/)
14. Kai Hong and Ani Nenkova. 2014. Improving the estimation of word importance for news multi-document summarization. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 712– 721. [find](https://scholar.google.com/scholar?q=Kai+Hong+and+Ani+Nenkova%2E+2014%2E+Improving+the+estimation+of+word+importance+for+news+multi%2Ddocument+summarization%2E+In+Proceedings+of+the+14th+Conference+of+the+European+Chapter+of+the+Association+for+)
15. Aikaterini-Lida Kalouli, Leo Vrana, Vigile Marie Fa-bella, Luna Bellani, and Annette Hautli-Janisz. 2018. Cousbi: A structured and visualized legal corpus of us state bills. In LREC 2018, pages 7–14. [find](https://scholar.google.com/scholar?q=Aikaterini%2DLida+Kalouli%2C+Leo+Vrana%2C+Vigile+Marie+Fa%2Dbella%2C+Luna+Bellani%2C+and+Annette+Hautli%2DJanisz%2E+2018%2E+Cousbi%3A+A+structured+and+visualized+legal+corpus+of+us+state+bills%2E+In+LREC+2018%2C+pages+7%E2%80%9314%2E)
16. Ambedkar Kanapala, Sukomal Pal, and Rajendra Pamula. 2017. Text summarization from legal documents: a survey. Artificial Intelligence Review, pages 1–32. [find](https://scholar.google.com/scholar?q=Ambedkar+Kanapala%2C+Sukomal+Pal%2C+and+Rajendra+Pamula%2E+2017%2E+Text+summarization+from+legal+documents%3A+a+survey%2E+Artificial+Intelligence+Review%2C+pages+1%E2%80%9332%2E)
17. Mi-Young Kim, Ying Xu, and Randy Goebel. 2013. Summarization of legal texts with high cohesion and automatic compression rate. In New Frontiers in Artificial Intelligence, pages 190–204, Berlin, Heidelberg. Springer Berlin Heidelberg. [find](https://scholar.google.com/scholar?q=Mi%2DYoung+Kim%2C+Ying+Xu%2C+and+Randy+Goebel%2E+2013%2E+Summarization+of+legal+texts+with+high+cohesion+and+automatic+compression+rate%2E+In+New+Frontiers+in+Artificial+Intelligence%2C+pages+190%E2%80%93204%2C+Berlin%2C+Heide)
18. Anastassia Kornilova, Daniel Argyle, and Vlad Eidelman. 2018. Party matters: Enhancing legislative embeddings with author attributes for vote prediction. arXiv preprint arXiv:1805.08182. [↗](https://arxiv.org/abs/1805.08182)
19. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. CoRR, abs/1901.08746. [↗](https://arxiv.org/abs/1901.08746) [our card](https://md.research.elamur.ai/free/en/papers/lee-2019-biobert-pre-trained-biomedical-language/)
20. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. [find](https://scholar.google.com/scholar?q=Chin%2DYew+Lin%2E+2004%2E+Rouge%3A+A+package+for+automatic+evaluation+of+summaries%2E+Text+Summarization+Branches+Out%2E) [our card](https://md.research.elamur.ai/free/en/papers/lin-2004-rouge-package-automatic-evaluation-summaries/)
21. Fridolin Linder, Bruce A. Desmarais, Matthew Burgess, and Eugenia Giraudy. 2018. Text as policy: Measuring policy similarity through bill text reuse. SSRN. [find](https://scholar.google.com/scholar?q=Fridolin+Linder%2C+Bruce+A%2E+Desmarais%2C+Matthew+Burgess%2C+and+Eugenia+Giraudy%2E+2018%2E+Text+as+policy%3A+Measuring+policy+similarity+through+bill+text+reuse%2E+SSRN%2E)
22. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bringing order into text. In Proceedings of the 2004 conference on empirical methods in natural language processing. [find](https://scholar.google.com/scholar?q=Rada+Mihalcea+and+Paul+Tarau%2E+2004%2E+Textrank%3A+Bringing+order+into+text%2E+In+Proceedings+of+the+2004+conference+on+empirical+methods+in+natural+language+processing%2E)
23. Ani Nenkova and Amit Bagga. 2004. Facilitating email thread access by extractive summary generation. AMSTERDAM STUDIES IN THE THEORY AND HISTORY OF LINGUISTIC SCIENCE, page 287. 4. Iteratively adding bills to the dataset, skipping examples that were more than 96% similar to any bills already added. [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Amit+Bagga%2E+2004%2E+Facilitating+email+thread+access+by+extractive+summary+generation%2E+AMSTERDAM+STUDIES+IN+THE+THEORY+AND+HISTORY+OF+LINGUISTIC+SCIENCE%2C+page+287%2E+4%2E+Iteratively+adding+)
24. Ani Nenkova and Lucy Vanderwende. 2005. The impact of frequency on summarization. Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005, 101. [find](https://scholar.google.com/scholar?q=Ani+Nenkova+and+Lucy+Vanderwende%2E+2005%2E+The+impact+of+frequency+on+summarization%2E+Microsoft+Research%2C+Redmond%2C+Washington%2C+Tech%2E+Rep%2E+MSR%2DTR%2D2005%2C+101%2E)
25. Juan Ramos et al. 2003. Using tf-idf to determine word relevance in document queries. In Proceedings of the first instructional conference on machine learning, volume 242, pages 133–142. [find](https://scholar.google.com/scholar?q=Juan+Ramos+et+al%2E+2003%2E+Using+tf%2Didf+to+determine+word+relevance+in+document+queries%2E+In+Proceedings+of+the+first+instructional+conference+on+machine+learning%2C+volume+242%2C+pages+133%E2%80%93142%2E)
26. Murali Saravanan, Balaraman Ravindran, Assistant Professor, and Dr Raman. 2008. Automatic identification of rhetorical roles using conditional random fields for legal document summarization. J. Artif. Intell. Law. [find](https://scholar.google.com/scholar?q=Murali+Saravanan%2C+Balaraman+Ravindran%2C+Assistant+Professor%2C+and+Dr+Raman%2E+2008%2E+Automatic+identification+of+rhetorical+roles+using+conditional+random+fields+for+legal+document+summarization%2E+J%2E+Artif%2E)
27. Yohei Seki. 2002. Sentence extraction by tf/idf and position weighting from newspaper articles. [find](https://scholar.google.com/scholar?q=Yohei+Seki%2E+2002%2E+Sentence+extraction+by+tf%2Fidf+and+position+weighting+from+newspaper+articles%2E)
28. Simone Teufel and Marc Moens. 2002. Summarizing scientific articles: experiments with relevance and rhetorical status. Computational linguistics, 28(4):409–445. [find](https://scholar.google.com/scholar?q=Simone+Teufel+and+Marc+Moens%2E+2002%2E+Summarizing+scientific+articles%3A+experiments+with+relevance+and+rhetorical+status%2E+Computational+linguistics%2C+28%284%29%3A409%E2%80%93445%2E) [our card](https://md.research.elamur.ai/free/en/papers/teufel-2002-summarizing-scientific-articles-experiments-relevance/)
29. Tae Yano, Noah A Smith, and John D Wilkerson. 2012. Textual predictors of bill survival in congressional committees. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 793–802. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Tae+Yano%2C+Noah+A+Smith%2C+and+John+D+Wilkerson%2E+2012%2E+Textual+predictors+of+bill+survival+in+congressional+committees%2E+In+Proceedings+of+the+2012+Conference+of+the+North+American+Chapter+of+the+Associat)
30. Markus Zopf, Eneldo Loza Mencı́a, and Johannes Fürnkranz. 2018. Which scores to predict in sentence regression for text summarization? In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1782–1791. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Markus+Zopf%2C+Eneldo+Loza+Menc%C4%B1%CC%81a%2C+and+Johannes+Fu%CC%88rnkranz%2E+2018%2E+Which+scores+to+predict+in+sentence+regression+for+text+summarization%3F+In+Proceedings+of+the+2018+Conference+of+the+North+American+Chap)