---
title: "Truth of Varying Shades — paper by Rashkin et al. (2017)"
description: "The paper looks at whether the wording of a text reveals how trustworthy it is. 0.65 F1. 23 references, cited by 204 papers here."
authors: ["Hannah Rashkin", "Eunsol Choi", "Jin Yea Jang", "Svitlana Volkova", "Yejin Choi"]
year: 2017
venue: "Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing"
source: "https://aclanthology.org/D17-1317/"
doi: "10.18653/v1/d17-1317"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/rashkin-2017-truth-varying-shades-analyzing-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/rashkin-2017-truth-varying-shades-analyzing-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/) / Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking

# Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking

**Authors:** [Hannah Rashkin](https://md.research.elamur.ai/free/en/authors/hannah-rashkin/), [Eunsol Choi](https://md.research.elamur.ai/free/en/authors/eunsol-choi/), [Jin Yea Jang](https://md.research.elamur.ai/free/en/authors/jin-yea-jang/), [Svitlana Volkova](https://md.research.elamur.ai/free/en/authors/svitlana-volkova/) and [Yejin Choi](https://md.research.elamur.ai/free/en/authors/yejin-choi/) · **Year:** 2017 · **Venue:** [Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2017](https://md.research.elamur.ai/free/en/venues/emnlp/2017/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/D17-1317/) · [PDF](https://aclanthology.org/D17-1317.pdf) · [DOI 10.18653/v1/d17-1317](https://doi.org/10.18653/v1/d17-1317) · [OpenAlex](https://openalex.org/W2759820691)

**Citations:** 870 citations (OpenAlex, 2026-10-14) · [204](https://md.research.elamur.ai/free/en/papers/rashkin-2017-truth-varying-shades-analyzing-language/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper looks at whether the wording of a text reveals how trustworthy it is. It treats truth as graded rather than as a simple true/false split. The authors built a corpus contrasting trusted English Gigaword articles with articles from seven unreliable sites, grouped as satire, hoax, or propaganda. They compared the groups using stylistic lexicons: LIWC, a subjectivity lexicon, hedge words, and new intensifier lists compiled from Wiktionary.

Fake news uses more first- and second-person pronouns, superlatives, subjective words, modal adverbs, and hedges. Trusted news leans on numbers, money terms, comparatives, and assertive verbs. A MaxEnt classifier on tf-idf n-grams sorts articles into four reliability classes with 0.65 F1 when the test sources are unseen.

A second case study predicts PolitiFact's 6-point ratings for political quotes. This task proves much harder: the best test macro-F1 is about 0.22 for six classes and 0.56 for two. The work matters because it measures style-based reliability cues across several kinds of untrustworthy text, and the authors released all of its resources publicly.

## Key points
- **News corpus:** trusted articles come from Gigaword. Unreliable articles come from:
  - satire: The Onion, The Borowitz Report, Clickhole
  - hoax: American News, DC Gazette
  - propaganda: The Natural News, Activist Report
- **Statistical testing:** the reported feature ratios (fake vs. trusted) are significant at p \< 0.01, using a Welch t-test with Bonferroni correction.
  - Swear words (ratio 7.00) and second-person "you" (6.73) are most inflated in fake news.
  - Numbers (0.43), "hear" words (0.50), and money terms (0.57) are most typical of trusted news.
- **4-way news reliability classification:** a MaxEnt classifier with L2 regularization on tf-idf features (up to trigrams) scores 0.91 F1 on in-domain dev data. It drops to 0.65 on out-of-domain test sources, against a random baseline of 0.26.
- **PolitiFact setup:** quotes were split 2575/712/1074 for train/dev/test, with all quotes from one speaker kept in the same split.
- **PolitiFact models:** an LSTM (GloVe 100-dim initialization, 300-dim hidden state) was compared with MaxEnt and Naive Bayes.
  - Adding LIWC features lifts NB and MaxEnt to roughly the LSTM's level.
  - The same features do not help the LSTM.

## Interesting ideas
- **Two-axis taxonomy:** unreliable content is classified by author intent (whether they want to deceive) and by factual quality. This separates satire from propaganda and hoaxes, and the same scheme could organize misinformation datasets in other languages.
- **Source-disjoint splits:** news articles are split by outlet, and PolitiFact quotes by speaker. This stops models from scoring well by memorizing author style. The large gap between in-domain and out-of-domain results shows how much this choice matters for honest evaluation.
- **Ratio-of-averages lexicon profiling:** the authors compare per-document lexicon counts between classes and apply significance tests with multiple-comparison correction. This is a cheap, interpretable way to describe a genre before building classifiers.
- **Crowd-built lexicons:** intensifier lists (comparatives, superlatives, action, manner, and modal adverbs) were harvested from Wiktionary. This trick transfers well to lower-resource languages where curated stylistic lexicons are missing.
- **Probing a neural model with lexicon features:** concatenating LIWC features to model inputs tests whether a neural model already captures those signals. Here the features gave no gain for the LSTM, suggesting the information was redundant, while the same features gave substantial gains to linear models.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of Washington](https://md.research.elamur.ai/free/en/universities/university-of-washington/) — University, Seattle, United States · 263 works · 180,581 citations

Also on the byline: [Pacific Northwest National Laboratory](https://ror.org/05h992307).

📚 Shelves (7): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,086 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,889 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 744 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works · [🤝 Sociology and Political Science](https://md.research.elamur.ai/free/en/shelves/social-sciences/sociology-and-political-science/) · 221 works

## Cite

Cite as: Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova and Yejin Choi. 2017. Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{rashkin2017truth,
  title = {{Truth of Varying Shades: Analyzing Language in Fake News and Political Fact-Checking}},
  author = {Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova and Yejin Choi},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1317},
  url = {https://doi.org/10.18653/v1/d17-1317}
}
```

- [← previous NewsQA: A Machine Comprehension Dataset](https://md.research.elamur.ai/free/en/papers/trischler-2017-newsqa-machine-comprehension-dataset/)
- [next → Adversarial Learning for Neural Dialogue Generation](https://md.research.elamur.ai/free/en/papers/li-2017-adversarial-learning-neural-dialogue-generation/)
- [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O’Reilly Media. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+2009%2E+Natural+Language+Processing+with+Python%2E+O%E2%80%99Reilly+Media%2E)
2. Prakhar Biyani, Kostas Tsioutsiouliklis, and John Blackmer. 2016. ”8 amazing secrets for getting more clicks”: Detecting clickbaits in news streams using article informality. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI Press, pages 94–100. [find](https://scholar.google.com/scholar?q=Prakhar+Biyani%2C+Kostas+Tsioutsiouliklis%2C+and+John+Blackmer%2E+2016%2E+%E2%80%9D8+amazing+secrets+for+getting+more+clicks%E2%80%9D%3A+Detecting+clickbaits+in+news+streams+using+article+informality%2E+In+Proceedings+of+the+Thi)
3. David B. Buller and Judee K. Burgoon. 1996. Interpersonal deception theory. Communication Theory 6(3):203–242. https://doi.org/10.1111/j.1468-2885.1996.tb00127.x. — [doi:10.1111/j.1468-2885.1996.tb00127.x](https://doi.org/10.1111/j.1468-2885.1996.tb00127.x) [↗](https://doi.org/10.1111/j.1468-2885.1996.tb00127.x)
4. Eunsol Choi, Chenhao Tan, Lillian Lee, Cristian Danescu-Niculescu-Mizil, and Jennifer Spindel. 2012. [Hedge detection as a lens on framing in the gmo debates: A position paper](https://md.research.elamur.ai/free/en/papers/choi-2012-hedge-detection-as-lens-framing/). In Proceedings of the Workshop on Extra-Propositional Aspects of Meaning in Computational Linguistics. Association for Compu…
5. Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis M. Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015. Computational fact checking from knowledge networks. PLOS ONE 10(6):e0128193. https://doi.org/10.1371/journal.pone.0128193. of the 25th International Conference on World Wide We… — [doi:10.1371/journal.pone.0128193](https://doi.org/10.1371/journal.pone.0128193) [↗](https://doi.org/10.1371/journal.pone.0128193)
6. Charles G Lord, Lee Ross, and Mark R Lepper. 1979. Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology 37(11):2098–2109. [find](https://scholar.google.com/scholar?q=Charles+G+Lord%2C+Lee+Ross%2C+and+Mark+R+Lepper%2E+1979%2E+Biased+assimilation+and+attitude+polarization%3A+The+effects+of+prior+theories+on+subsequently+considered+evidence%2E+Journal+of+Personality+and+Social+P)
7. Rada Mihalcea and Carlo Strapparava. 2009. [The lie detector: Explorations in the automatic recognition of deceptive language](https://md.research.elamur.ai/free/en/papers/mihalcea-2009-lie-detector-explorations-automatic-recognition/). In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers. Association for Computational Linguistics, pages 309–312.
8. Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards. 2003. Lying words: Predicting deception from linguistic styles. Personality and social psychology bulletin 29(5):665–675. [find](https://scholar.google.com/scholar?q=Matthew+L+Newman%2C+James+W+Pennebaker%2C+Diane+S+Berry%2C+and+Jane+M+Richards%2E+2003%2E+Lying+words%3A+Predicting+deception+from+linguistic+styles%2E+Personality+and+social+psychology+bulletin+29%285%29%3A665%E2%80%93675%2E)
9. Brendan Nyhan and Jason Reifler. 2015. The effect of fact-checking on elites: A field experiment on US state legislators. American Journal of Political Science 59(3):628–640. [find](https://scholar.google.com/scholar?q=Brendan+Nyhan+and+Jason+Reifler%2E+2015%2E+The+effect+of+fact%2Dchecking+on+elites%3A+A+field+experiment+on+US+state+legislators%2E+American+Journal+of+Political+Science+59%283%29%3A628%E2%80%93640%2E)
10. Myle Ott, Yejin Choi, Claire Cardie, and Jeffrey T. Hancock. 2011. [Finding deceptive opinion spam by any stretch of the imagination](https://md.research.elamur.ai/free/en/papers/ott-2011-finding-deceptive-opinion-spam-any/). In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, pag…
11. Kate Connolly, Angelique Chrisafis, Poppy McPherson, Stephanie Kirchgaessner, Benjamin Haas, Dominic Phillips, Elle Hunt, and James W. Pennebaker, Roger J. Booth, Ryan L. Boyd Michael Safi. 2016. Fake news: An insidi-Boyd, and Martha E. Francis. 2015. Linguistic In-ous trend that’s fast becoming a g… [find](https://scholar.google.com/scholar?q=Kate+Connolly%2C+Angelique+Chrisafis%2C+Poppy+McPherson%2C+Stephanie+Kirchgaessner%2C+Benjamin+Haas%2C+Dominic+Phillips%2C+Elle+Hunt%2C+and+James+W%2E+Pennebaker%2C+Roger+J%2E+Booth%2C+Ryan+L%2E+Boyd+Michael+Safi%2E+2016%2E+Fake)
12. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word Codruta Girlea, Roxana Girju, and Eyal Amir. 2016. representation. In Proceedings of the 2014 Con-Psycholinguistic features for deceptive role detec-ference on Empirical Methods in Natural Language tion… [find](https://scholar.google.com/scholar?q=Jeffrey+Pennington%2C+Richard+Socher%2C+and+Christopher+Manning%2E+2014%2E+Glove%3A+Global+vectors+for+word+Codruta+Girlea%2C+Roxana+Girju%2C+and+Eyal+Amir%2E+2016%2E+representation%2E+In+Proceedings+of+the+2014+Con%2DPsyc)
13. Ken Hyland. 2015. Metadiscourse. In The International Encyclopedia of Language and Social Interaction, John Wiley & Sons, Inc., pages 1–11. [find](https://scholar.google.com/scholar?q=Ken+Hyland%2E+2015%2E+Metadiscourse%2E+In+The+International+Encyclopedia+of+Language+and+Social+Interaction%2C+John+Wiley+%26+Sons%2C+Inc%2E%2C+pages+1%E2%80%9311%2E)
14. Nitin Jindal and Bing Liu. 2008. Opinion spam and analysis. In Proceedings of the 2008 International Conference on Web Search and Data Mining. ACM, pages 219–230. [find](https://scholar.google.com/scholar?q=Nitin+Jindal+and+Bing+Liu%2E+2008%2E+Opinion+spam+and+analysis%2E+In+Proceedings+of+the+2008+International+Conference+on+Web+Search+and+Data+Mining%2E+ACM%2C+pages+219%E2%80%93230%2E)
15. Srijan Kumar, Robert West, and Jure Leskovec. 2016. Disinformation on the web: Impact, characteristics, and detection of wikipedia hoaxes. In Proceedings Paul Rayson, Andrew Wilson, and Geoffrey Leech. 2001. Grammatical word class variation within the british national corpus sampler. Language and Co… [find](https://scholar.google.com/scholar?q=Srijan+Kumar%2C+Robert+West%2C+and+Jure+Leskovec%2E+2016%2E+Disinformation+on+the+web%3A+Impact%2C+characteristics%2C+and+detection+of+wikipedia+hoaxes%2E+In+Proceedings+Paul+Rayson%2C+Andrew+Wilson%2C+and+Geoffrey+Leech)
16. Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013. [Linguistic models for analyzing and detecting biased language](https://md.research.elamur.ai/free/en/papers/recasens-2013-linguistic-models-analyzing-detecting-biased/). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, page…
17. Victoria L. Rubin, Yimin Chen, and Niall J. Conroy. 2015. Deception detection for news: Three types of fakes. Proceedings of the Association for Information Science and Technology 52(1):1–4. [find](https://scholar.google.com/scholar?q=Victoria+L%2E+Rubin%2C+Yimin+Chen%2C+and+Niall+J%2E+Conroy%2E+2015%2E+Deception+detection+for+news%3A+Three+types+of+fakes%2E+Proceedings+of+the+Association+for+Information+Science+and+Technology+52%281%29%3A1%E2%80%934%2E)
18. Emily Thorson. 2016. Belief echoes: The persistent effects of corrected misinformation. Political Communication 33(3):460–480. [find](https://scholar.google.com/scholar?q=Emily+Thorson%2E+2016%2E+Belief+echoes%3A+The+persistent+effects+of+corrected+misinformation%2E+Political+Communication+33%283%29%3A460%E2%80%93480%2E)
19. Andreas Vlachos and Sebastian Riedel. 2014. [Fact checking: Task definition and dataset construction](https://md.research.elamur.ai/free/en/papers/vlachos-2014-fact-checking-task-definition-dataset/). In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science. Association for Computational Linguistics, pages 18–22.
20. William Yang Wang. 2017. [“Liar, liar pants on fire”: A new benchmark dataset for fake news detection](https://md.research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/). In Proceedings of the Association for Computational Linguistics Short Papers. Association for Computational Linguistics.
21. Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. [Recognizing contextual polarity in phrase-level sentiment analysis](https://md.research.elamur.ai/free/en/papers/wilson-2005-recognizing-contextual-polarity-phrase-level/). In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 347–3…
22. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. [Character-level convolutional networks for text classification](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/). In Advances in Neural Information Processing Systems. pages 649–657.
23. Lina Zhou, Judee K. Burgoon, Jay F. Nunamaker, and Doug Twitchell. 2004. Automating linguistics-based cues for detecting deception in text-based asynchronous computer-mediated communications. Group Decision and Negotiation 13(1):81–106. [find](https://scholar.google.com/scholar?q=Lina+Zhou%2C+Judee+K%2E+Burgoon%2C+Jay+F%2E+Nunamaker%2C+and+Doug+Twitchell%2E+2004%2E+Automating+linguistics%2Dbased+cues+for+detecting+deception+in+text%2Dbased+asynchronous+computer%2Dmediated+communications%2E+Group+De)

*Updated 24 October 2026*