---
title: "The Media Frames Corpus — Card 2015"
description: "Social scientists usually study media framing with codebooks built for a single issue."
authors: ["Dallas Card", "Amber E. Boydstun", "Justin H. Gross", "Philip Resnik", "Noah A. Smith"]
year: 2015
venue: "Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)"
source: "https://aclanthology.org/P15-2072/"
doi: "10.3115/v1/p15-2072"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/card-2015-media-frames-corpus-annotations-frames/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/card-2015-media-frames-corpus-annotations-frames/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/) / The Media Frames Corpus: Annotations of Frames Across Issues

# The Media Frames Corpus: Annotations of Frames Across Issues

**Authors:** [Dallas Card](https://md.research.elamur.ai/free/en/authors/dallas-card/), Amber E. Boydstun, Justin H. Gross, [Philip Resnik](https://md.research.elamur.ai/free/en/authors/philip-resnik/), [Noah A. Smith](https://md.research.elamur.ai/free/en/authors/noah-a-smith/) · **Year:** 2015 · **Venue:** Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/P15-2072/) · [PDF](https://aclanthology.org/P15-2072.pdf) · [DOI 10.3115/v1/p15-2072](https://doi.org/10.3115/v1/p15-2072) · [OpenAlex](https://openalex.org/W2251617662)

**Citations:** 219 citations (openalex-snapshot · 2026-10-14) · [140](#cited-by) in our library, of the 140884 papers whose bibliographies we read

## Summary
Social scientists usually study media framing with codebooks built for a single issue. This makes it hard to compare framing across topics, and earlier computational work mostly reduced framing to one dimension or one domain. The authors introduce the Media Frames Corpus (version 1), which contains several thousand U.S. newspaper articles on immigration, smoking, and same-sex marriage. The articles are annotated with 15 issue-independent "framing dimensions" from Boydstun et al. (2014), such as Economic, Morality, and Political. Undergraduate annotators tagged every dimension present in an article, highlighted the text spans that cue each one (spans may overlap), and chose a primary frame for the headline and for the whole article. Annotation proceeded in three stages, and multiple coding increased over time. Agreement was measured with Krippendorff's α on primary frames and with its unitizing variant α_U on spans. Span-level agreement is low but consistently above chance, and agreement rose as annotators gained experience. The annotations are released publicly, giving NLP researchers a shared resource for modeling framing across policy issues.

## Key points
- The articles were retrieved via Lexis-Nexis from 13 national U.S. newspapers published between 1990 and 2012. Immigration coverage goes back to 1969, though few articles predate 1980. Duplicates and near-duplicates were removed before random sampling.
- The annotation was done in three stages, with the average number of annotators per article shown in parentheses:
  - Stage 1: 4,113 immigration and 4,077 smoking articles (about 1.2 annotators each).
  - Stage 2: 6,298 same-sex marriage articles (about 2.2 annotators each).
  - Stage 3: 5,549 immigration articles (about 2.2 annotators each).
- In total, 19 undergraduate annotators took part. Across annotators, the average number of frames per article ranged from 2.0 to 3.7, and the average number of highlighted spans ranged from 3.4 to 10.0. The "Other" dimension appeared in fewer than 10% of cases.
- Spans reached an average α_U of 0.16 in the stage 1 immigration articles, 0.23 in the tobacco articles, 0.08 in the same-sex marriage articles, and 0.20 in the stage 3 immigration articles.
- Annotations, metadata, and tools for re-acquiring the licensed texts are available at github.com/dallascard/media_frames_corpus.

## Interesting ideas
- **A shared label set across topics.** Using a single cross-cutting label set instead of issue-specific codebooks makes results comparable across topics. The same approach could suit any multi-domain annotation effort, for example tracking how Ukrainian media frame trade versus security.
- **Disagreement as signal.** The authors treat some annotator disagreement as legitimate differences in interpretation rather than noise, and they keep multiple annotations instead of forcing a gold label. This matters for subjective tasks, where collapsing labels to one answer can erase real variation.
- **Documenting how the data was made.** The release openly preserves process effects, such as annotators improving with practice and guidelines changing over time. Recording these effects alongside the data lets downstream users account for drift instead of assuming a uniform gold standard.
- **Agreement for spans with fuzzy boundaries.** α_U scores span agreement using the squared lengths of non-overlapping segments, and it is reported together with a simpler document-level α. Pairing a strict span metric with a coarse label metric gives a more honest picture for unitizing tasks such as NER or argument mining.
- **Publishing under license constraints.** Rather than distributing copyrighted texts, the authors release annotations and offsets plus scripts that rebuild the corpus for anyone holding a license. This is a practical model for sharing news-based datasets.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Cited by

Cited by

140 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 135 not yet.

- [The Pragmatics behind Politics: Modelling Metaphor, Framing and Emotion in Political Discourse](https://md.research.elamur.ai/free/en/papers/cabot-2020-pragmatics-behind-politics-modelling-metaphor/)
- [HeBID: Detecting Social Identities in Hebrew-language Political Text](https://md.research.elamur.ai/free/en/papers/findings-2025-lan-hebid-detecting-social-identities-hebrew/)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/)
- [Frame In, Frame Out: Measuring Framing Bias in LLM-Generated News Summaries](https://md.research.elamur.ai/free/en/papers/pastorino-2026-frame-frame-out-measuring-framing/)
- [From Shakespeare to Twitter: What are Language Styles all about?](https://md.research.elamur.ai/free/en/papers/xu-2017-shakespeare-twitter-what-are-language/)

## Cite

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{card2015media,
  title = {{The Media Frames Corpus: Annotations of Frames Across Issues}},
  author = {Card, Dallas and Boydstun, Amber E. and Gross, Justin H. and Resnik, Philip and Smith, Noah A.},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-2072},
  url = {https://doi.org/10.3115/v1/p15-2072}
}
```

- [← previous The Hebrew Bible as Data: Laboratory – Sharing – Experiences](https://md.research.elamur.ai/free/en/papers/hbo-2015-roorda-hebrew-bible-data-laboratory-sharing/)
- [next → The Multidimensional Quality Metrics (MQM) Framework: a new framework for translation quality assessment](https://md.research.elamur.ai/free/en/papers/mariana-2015-multidimensional-quality-metrics-mqm-framework/)
- [↑ back to papers of 2015](https://md.research.elamur.ai/free/en/papers/2015/page/2/)

## References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amr Ahmed and Eric P. Xing. 2010. Staying informed: Supervised and semi-supervised multi-view topical analysis of ideological perspective. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Amr+Ahmed+and+Eric+P%2E+Xing%2E+2010%2E+Staying+informed%3A+Supervised+and+semi%2Dsupervised+multi%2Dview+topical+analysis+of+ideological+perspective%2E+In+Proc%2E+of+EMNLP%2E)
2. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. [find](https://scholar.google.com/scholar?q=Ron+Artstein+and+Massimo+Poesio%2E+2008%2E+Inter%2Dcoder+agreement+for+computational+linguistics%2E+Computational+Linguistics%2C+34%284%29%3A555%E2%80%93596%2E)
3. Eric Baumer, Elisha Elovic, Ying Qin, Francesca Polletta, and Geri Gay. 2015. Testing and comparing computational approaches for identifying the language of framing in political news. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Eric+Baumer%2C+Elisha+Elovic%2C+Ying+Qin%2C+Francesca+Polletta%2C+and+Geri+Gay%2E+2015%2E+Testing+and+comparing+computational+approaches+for+identifying+the+language+of+framing+in+political+news%2E+In+Proc%2E+of+NAAC)
4. Frank R. Baumgartner, Suzanna L. De Boef, and Amber E. Boydstun. 2008. The decline of the death penalty and the discovery of innocence. Cambridge University Press. [find](https://scholar.google.com/scholar?q=Frank+R%2E+Baumgartner%2C+Suzanna+L%2E+De+Boef%2C+and+Amber+E%2E+Boydstun%2E+2008%2E+The+decline+of+the+death+penalty+and+the+discovery+of+innocence%2E+Cambridge+University+Press%2E)
5. Robert D. Benford and David A. Snow. 2000. Framing processes and social movements: An overview and assessment. Annual Review of Sociology, 26:611– 639. [find](https://scholar.google.com/scholar?q=Robert+D%2E+Benford+and+David+A%2E+Snow%2E+2000%2E+Framing+processes+and+social+movements%3A+An+overview+and+assessment%2E+Annual+Review+of+Sociology%2C+26%3A611%E2%80%93+639%2E)
6. Adam J. Berinsky and Donald R. Kinder. 2006. Making sense of issues through media frames: Understanding the Kosovo crisis. Journal of Politics, 68(3):640–656. [find](https://scholar.google.com/scholar?q=Adam+J%2E+Berinsky+and+Donald+R%2E+Kinder%2E+2006%2E+Making+sense+of+issues+through+media+frames%3A+Understanding+the+Kosovo+crisis%2E+Journal+of+Politics%2C+68%283%29%3A640%E2%80%93656%2E)
7. Amber E. Boydstun, Dallas Card, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2014. Tracking the development of media frames within and across policy issues. [find](https://scholar.google.com/scholar?q=Amber+E%2E+Boydstun%2C+Dallas+Card%2C+Justin+H%2E+Gross%2C+Philip+Resnik%2C+and+Noah+A%2E+Smith%2E+2014%2E+Tracking+the+development+of+media+frames+within+and+across+policy+issues%2E)
8. APSA 2014 Annual Meeting Paper. [find](https://scholar.google.com/scholar?q=APSA+2014+Annual+Meeting+Paper%2E)
9. Eunsol Choi, Chenhao Tan, Lillian Lee, Cristian Danescu-Niculescu-Mizil, and Jennifer Spindel. 2012. Hedge detection as a lens on framing in the GMO debates: A position paper. In Proc of. ACL Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics, pages 70–79. [find](https://scholar.google.com/scholar?q=Eunsol+Choi%2C+Chenhao+Tan%2C+Lillian+Lee%2C+Cristian+Danescu%2DNiculescu%2DMizil%2C+and+Jennifer+Spindel%2E+2012%2E+Hedge+detection+as+a+lens+on+framing+in+the+GMO+debates%3A+A+position+paper%2E+In+Proc+of%2E+ACL+Workshop)
10. Dennis Chong and James N. Druckman. 2007. Framing theory. Annual Review of Political Science, 10(1):103–126. [find](https://scholar.google.com/scholar?q=Dennis+Chong+and+James+N%2E+Druckman%2E+2007%2E+Framing+theory%2E+Annual+Review+of+Political+Science%2C+10%281%29%3A103%E2%80%93126%2E)
11. Robert M. Entman. 1993. Framing: Toward clarification of a fractured paradigm. Journal of Communication, 43(4):51–58. [find](https://scholar.google.com/scholar?q=Robert+M%2E+Entman%2E+1993%2E+Framing%3A+Toward+clarification+of+a+fractured+paradigm%2E+Journal+of+Communication%2C+43%284%29%3A51%E2%80%9358%2E)
12. William A. Gameson and Andre Modigliani. 1989. Media discourse and public opinion on nuclear power: A constructionist approach. American Journal of Sociology, 95(1):1–37. [find](https://scholar.google.com/scholar?q=William+A%2E+Gameson+and+Andre+Modigliani%2E+1989%2E+Media+discourse+and+public+opinion+on+nuclear+power%3A+A+constructionist+approach%2E+American+Journal+of+Sociology%2C+95%281%29%3A1%E2%80%9337%2E)
13. Matthew Gentzkow and Jesse M. Shapiro. 2010. What drives media slant? Evidence from U.S. daily newspapers. Econometrica, 78(1):35–71. [find](https://scholar.google.com/scholar?q=Matthew+Gentzkow+and+Jesse+M%2E+Shapiro%2E+2010%2E+What+drives+media+slant%3F+Evidence+from+U%2ES%2E+daily+newspapers%2E+Econometrica%2C+78%281%29%3A35%E2%80%9371%2E)
14. Eric Hardisty, Jordan L. Boyd-Graber, and Philip Resnik. 2010. Modeling perspective using adaptor grammars. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Eric+Hardisty%2C+Jordan+L%2E+Boyd%2DGraber%2C+and+Philip+Resnik%2E+2010%2E+Modeling+perspective+using+adaptor+grammars%2E+In+Proc%2E+of+EMNLP%2E)
15. Kazi Saidul Hasan and Vincent Ng. 2013. Stance classification of ideological debates: Data, models, features, and constraints. In Proc. of IJCNLP. [find](https://scholar.google.com/scholar?q=Kazi+Saidul+Hasan+and+Vincent+Ng%2E+2013%2E+Stance+classification+of+ideological+debates%3A+Data%2C+models%2C+features%2C+and+constraints%2E+In+Proc%2E+of+IJCNLP%2E)
16. Mohit Iyyer, Peter Enns, Jordan L. Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Mohit+Iyyer%2C+Peter+Enns%2C+Jordan+L%2E+Boyd%2DGraber%2C+and+Philip+Resnik%2E+2014%2E+Political+ideology+detection+using+recursive+neural+networks%2E+In+Proc%2E+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/iyyer-2014-political-ideology-detection-recursive-neural/)
17. Beata Beigman Klebanov, Eyal Beigman, and Daniel Diermeier. 2010. Vocabulary choice as an indicator of perspective. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Beata+Beigman+Klebanov%2C+Eyal+Beigman%2C+and+Daniel+Diermeier%2E+2010%2E+Vocabulary+choice+as+an+indicator+of+perspective%2E+In+Proc%2E+of+ACL%2E)
18. Klaus Krippendorff. 2004. Measuring the reliability of qualitative text analysis data. Quality and Quantity, 38(6):787–800. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2004%2E+Measuring+the+reliability+of+qualitative+text+analysis+data%2E+Quality+and+Quantity%2C+38%286%29%3A787%E2%80%93800%2E)
19. Klaus Krippendorff. 2012. Content Analysis: An Introduction to its Methodology. SAGE Publications. [find](https://scholar.google.com/scholar?q=Klaus+Krippendorff%2E+2012%2E+Content+Analysis%3A+An+Introduction+to+its+Methodology%2E+SAGE+Publications%2E)
20. Jure Leskovec, Lars Backstrom, and Jon Kleinberg. 2009. Meme-tracking and the dynamics of the news cycle. In Proc. of KDD. [find](https://scholar.google.com/scholar?q=Jure+Leskovec%2C+Lars+Backstrom%2C+and+Jon+Kleinberg%2E+2009%2E+Meme%2Dtracking+and+the+dynamics+of+the+news+cycle%2E+In+Proc%2E+of+KDD%2E)
21. Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006. Which side are you on? Identifying perspectives at the document and sentence levels. In Proc. of CoNNL. [find](https://scholar.google.com/scholar?q=Wei%2DHao+Lin%2C+Theresa+Wilson%2C+Janyce+Wiebe%2C+and+Alexander+Hauptmann%2E+2006%2E+Which+side+are+you+on%3F+Identifying+perspectives+at+the+document+and+sentence+levels%2E+In+Proc%2E+of+CoNNL%2E)
22. Viet-An Nguyen, Jordan Boyd-Graber, Philip Resnik, and Kristina Miler. 2015. Tea party in the house: A hierarchical ideal point topic model and its application to Republican legislators in the 112th congress. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Viet%2DAn+Nguyen%2C+Jordan+Boyd%2DGraber%2C+Philip+Resnik%2C+and+Kristina+Miler%2E+2015%2E+Tea+party+in+the+house%3A+A+hierarchical+ideal+point+topic+model+and+its+application+to+Republican+legislators+in+the+112th+c)
23. Zhongdang Pan and Gerald M. Kosicki. 1993. Framing analysis: An approach to news discourse. Political communication, 10(1):55–75. [find](https://scholar.google.com/scholar?q=Zhongdang+Pan+and+Gerald+M%2E+Kosicki%2E+1993%2E+Framing+analysis%3A+An+approach+to+news+discourse%2E+Political+communication%2C+10%281%29%3A55%E2%80%9375%2E)
24. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Rebecca+J%2E+Passonneau+and+Bob+Carpenter%2E+2014%2E+The+benefits+of+a+model+of+annotation%2E+In+Proc%2E+of+ACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/passonneau-2014-benefits-model-annotation/)
25. Stephen D. Rees, Oscar H. Gandy Jr., , and August E. Grant, editors. 2001. Framing public life: Perspectives on media and our understanding of the social world. Routledge. [find](https://scholar.google.com/scholar?q=Stephen+D%2E+Rees%2C+Oscar+H%2E+Gandy+Jr%2E%2C+%2C+and+August+E%2E+Grant%2C+editors%2E+2001%2E+Framing+public+life%3A+Perspectives+on+media+and+our+understanding+of+the+social+world%2E+Routledge%2E)
26. Yanchuan Sim, Brice D. L. Acree, Justin H. Gross, and Noah A. Smith. 2013. Measuring ideological proportions in political speeches. In Proc. of EMNLP. [find](https://scholar.google.com/scholar?q=Yanchuan+Sim%2C+Brice+D%2E+L%2E+Acree%2C+Justin+H%2E+Gross%2C+and+Noah+A%2E+Smith%2E+2013%2E+Measuring+ideological+proportions+in+political+speeches%2E+In+Proc%2E+of+EMNLP%2E)
27. David A. Smith, Ryan Cordell, and Elizabeth Maddock Dillon. 2013. Infectious texts: modeling text reuse in nineteenth-century newspapers. In IEEE International Conference on Big Data. [find](https://scholar.google.com/scholar?q=David+A%2E+Smith%2C+Ryan+Cordell%2C+and+Elizabeth+Maddock+Dillon%2E+2013%2E+Infectious+texts%3A+modeling+text+reuse+in+nineteenth%2Dcentury+newspapers%2E+In+IEEE+International+Conference+on+Big+Data%2E)
28. David A. Smith, Ryan Cordel, Elizabeth Maddock Dillon, Nick Stramp, and John Wilkerson. 2014. Detecting and modeling local text reuse. In IEEE/ACM Joint Conference on Digital Libraries. [find](https://scholar.google.com/scholar?q=David+A%2E+Smith%2C+Ryan+Cordel%2C+Elizabeth+Maddock+Dillon%2C+Nick+Stramp%2C+and+John+Wilkerson%2E+2014%2E+Detecting+and+modeling+local+text+reuse%2E+In+IEEE%2FACM+Joint+Conference+on+Digital+Libraries%2E)
29. Nayda Terkildsen and Frauke Schnell. 1997. How media frames move public opinion: An analysis of the women’s movement. Political research quarterly, 50(4):879–900. [find](https://scholar.google.com/scholar?q=Nayda+Terkildsen+and+Frauke+Schnell%2E+1997%2E+How+media+frames+move+public+opinion%3A+An+analysis+of+the+women%E2%80%99s+movement%2E+Political+research+quarterly%2C+50%284%29%3A879%E2%80%93900%2E)
30. Oren Tsur, Dan Calacci, and David Lazer. 2015. Frame of mind: Using statistical models for detection of framing and agenda setting campaigns. In Proc. of ACL. [find](https://scholar.google.com/scholar?q=Oren+Tsur%2C+Dan+Calacci%2C+and+David+Lazer%2E+2015%2E+Frame+of+mind%3A+Using+statistical+models+for+detection+of+framing+and+agenda+setting+campaigns%2E+In+Proc%2E+of+ACL%2E)
31. Marilyn A. Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proc. of NAACL. [find](https://scholar.google.com/scholar?q=Marilyn+A%2E+Walker%2C+Pranav+Anand%2C+Robert+Abbott%2C+and+Ricky+Grant%2E+2012%2E+Stance+classification+using+dialogic+properties+of+persuasion%2E+In+Proc%2E+of+NAACL%2E) [our card](https://md.research.elamur.ai/free/en/papers/walker-2012-stance-classification-dialogic-properties-persuasion/)
32. Janyce Wiebe, Theresa Wilson, Rebecca Bruce, Matthew Bell, and Melanie Martin. 2004. Learning subjective language. Computational Linguistics, 30(3):277–308. [find](https://scholar.google.com/scholar?q=Janyce+Wiebe%2C+Theresa+Wilson%2C+Rebecca+Bruce%2C+Matthew+Bell%2C+and+Melanie+Martin%2E+2004%2E+Learning+subjective+language%2E+Computational+Linguistics%2C+30%283%29%3A277%E2%80%93308%2E) [our card](https://md.research.elamur.ai/free/en/papers/wiebe-2004-learning-subjective-language/)