--- title: "“Liar, Liar Pants on Fire” — Wang 2017" description: "Research on detecting fake news by statistical methods has been held back by the lack of large labeled benchmarks." authors: ["William Yang Wang"] year: 2017 venue: "Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)" source: "https://aclanthology.org/P17-2067/" doi: "10.18653/v1/p17-2067" licence: "CC BY 4.0 (ACL Anthology)" licence_url: "https://creativecommons.org/licenses/by/4.0/" summary_licence: "Elamur Cosmic License" canonical: "https://research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/" --- [Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/wang-2017-liar-liar-pants-fire-new/) [home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) / “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection # “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection **Authors:** [William Yang Wang](https://md.research.elamur.ai/free/en/authors/william-yang-wang/) · **Year:** 2017 · **Venue:** Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/) **Links:** [Publisher's page](https://aclanthology.org/P17-2067/) · [PDF](https://aclanthology.org/P17-2067.pdf) · [DOI 10.18653/v1/p17-2067](https://doi.org/10.18653/v1/p17-2067) · [OpenAlex](https://openalex.org/W2611921107) **Citations:** 1516 citations (openalex-snapshot · 2026-10-14) · [285](#cited-by) in our library, of the 140673 papers whose bibliographies we read ## Summary Research on detecting fake news by statistical methods has been held back by the lack of large labeled benchmarks. Earlier public collections had only a few hundred political claims, which is too few for machine learning. This paper introduces LIAR, a public dataset of 12,836 short political statements gathered through the POLITIFACT.COM API. The statements date mainly from 2007–2016, and PolitiFact editors rated each one on a six-level truthfulness scale. Each statement also comes with speaker metadata, its context and subject, and an analysis report with links to sources. The author treats detection as six-way text classification. Baselines are logistic regression, SVMs, Bi-LSTMs and a Kim-style CNN. The paper also proposes a hybrid CNN that encodes metadata with convolution and a bi-directional LSTM, then combines it with the text representation. Of the text-only models, the CNN scores best. Adding all metadata raises test accuracy further. LIAR is roughly ten times larger than earlier comparable resources and draws on real-world sources rather than crowdsourcing. It can also support fact-checking, stance and rumor research. ## Key points - **Dataset:** LIAR contains 12,836 statements, split into 10,269 train, 1,284 validation and 1,283 test. The average statement is 17.9 tokens long. It is downloadable from the author's UCSB page. - **Labels:** the six ratings are pants-fire, false, barely-true, half-true, mostly-true and true. The classes are fairly balanced: pants-fire has 1,050 cases and each other label has 2,063–2,638. On a sample, agreement with the PolitiFact analyses reached Cohen's kappa 0.82. - **Metadata:** each statement includes subject, speaker, job, home state, party, venue and a "credit history" vector. That vector counts the speaker's past ratings and must exclude the current label to avoid leakage. - **Text-only test accuracy:** majority baseline 0.208, LR 0.247, SVM 0.255, Bi-LSTM 0.233 (it overfits), CNN 0.270. The CNN's advantage over the SVM is significant (p \< .0001). - **Hybrid model:** the hybrid CNN using text plus all metadata reaches about 0.274 test accuracy, the best result in the paper. The extracted table was garbled, so check this figure and the per-feature results against the original paper. *Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.* Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/) ## Cited by Cited by 285 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 265 not yet. - [MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims](https://md.research.elamur.ai/free/en/papers/augenstein-2019-multifc-real-world-multi-domain/) - [Integrating Stance Detection and Fact Checking in a Unified Corpus](https://md.research.elamur.ai/free/en/papers/baly-2018-integrating-stance-detection-fact-checking/) - [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://md.research.elamur.ai/free/en/papers/chen-2019-tabfact-large-scale-dataset-table/) - [From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models](https://md.research.elamur.ai/free/en/papers/feng-2023-pretraining-data-language-models-downstream/) - [The Truth, The Whole Truth, and Nothing but the Truth: A New Benchmark Dataset for Hebrew Text Credibility Assessment](https://md.research.elamur.ai/free/en/papers/findings-2023-hagag-truth-whole-truth-nothing-but/) - [RARR: Researching and Revising What Language Models Say, Using Language Models](https://md.research.elamur.ai/free/en/papers/gao-2023-rarr-researching-revising-what-language/) - [GLTR: Statistical Detection and Visualization of Generated Text](https://md.research.elamur.ai/free/en/papers/gehrmann-2019-gltr-statistical-detection-visualization-generated/) - [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://md.research.elamur.ai/free/en/papers/gou-2023-critic-large-language-models-self/) - [A Survey on Automated Fact-Checking](https://md.research.elamur.ai/free/en/papers/guo-2022-survey-automated-fact-checking/) - [A Survey on Stance Detection for Mis- and Disinformation Identification](https://md.research.elamur.ai/free/en/papers/hardalov-2022-survey-stance-detection-mis-disinformation/) - [HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification](https://md.research.elamur.ai/free/en/papers/jiang-2020-hover-dataset-many-hop-fact/) - [Explainable Automated Fact-Checking for Public Health Claims](https://md.research.elamur.ai/free/en/papers/kotonya-2020-explainable-automated-fact-checking-public/) - [A Stylometric Inquiry into Hyperpartisan and Fake News](https://md.research.elamur.ai/free/en/papers/potthast-2018-stylometric-inquiry-into-hyperpartisan-fake/) - [Automatic Prompt Optimization with “Gradient Descent” and Beam Search](https://md.research.elamur.ai/free/en/papers/pryzant-2023-automatic-prompt-optimization-gradient-descent/) - [Automatic Detection of Fake News](https://md.research.elamur.ai/free/en/papers/rosas-2018-automatic-detection-fake-news/) - [Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence](https://md.research.elamur.ai/free/en/papers/schuster-2021-get-your-vitamin-c-robust/) - [FEVER: a Large-scale Dataset for Fact Extraction and VERification](https://md.research.elamur.ai/free/en/papers/thorne-2018-fever-large-scale-dataset-fact/) - [Fact or Fiction: Verifying Scientific Claims](https://md.research.elamur.ai/free/en/papers/wadden-2020-fact-or-fiction-verifying-scientific/) - [TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification](https://md.research.elamur.ai/free/en/papers/yin-2018-twowingos-two-wing-optimization-strategy/) - [Defending Against Neural Fake News](https://md.research.elamur.ai/free/en/papers/zellers-2019-defending-against-neural-fake-news/) ## Cite The paper itself, not this page. BibTeX from the library's record of the work. ```bibtex @inproceedings{wang2017liar, title = {{“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection}}, author = {Wang, William Yang}, booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)}, year = {2017}, doi = {10.18653/v1/p17-2067}, url = {https://doi.org/10.18653/v1/p17-2067} } ``` - [← previous Word Sense Disambiguation: A Unified Evaluation Framework and Empirical Comparison](https://md.research.elamur.ai/free/en/papers/onto-2017-raganato-word-sense-disambiguation-unified-evaluation/) - [↑ back to papers of 2017](https://md.research.elamur.ai/free/en/papers/2017/page/3/) ## References 11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet. 1. Verónica Pérez-Rosas and Rada Mihalcea. 2015. Experiments in open domain deception detection. In EMNLP. pages 1120–1125. [find](https://scholar.google.com/scholar?q=Vero%CC%81nica+Pe%CC%81rez%2DRosas+and+Rada+Mihalcea%2E+2015%2E+Experiments+in+open+domain+deception+detection%2E+In+EMNLP%2E+pages+1120%E2%80%931125%2E) 2. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12(Aug):2493–2537. [find](https://scholar.google.com/scholar?q=Ronan+Collobert%2C+Jason+Weston%2C+Le%CC%81on+Bottou%2C+Michael+Karlen%2C+Koray+Kavukcuoglu%2C+and+Pavel+Kuksa%2E+2011%2E+Natural+language+processing+%28almost%29+from+scratch%2E+Journal+of+Machine+Learning+Research+12%28Aug%29%3A2) [our card](https://md.research.elamur.ai/free/en/papers/collobert-2011-natural-language-processing-almost-scratch/) 3. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. Proceedings of the ACL 2014 Workshop on Language Technology and Computational Social Science . [find](https://scholar.google.com/scholar?q=Andreas+Vlachos+and+Sebastian+Riedel%2E+2014%2E+Fact+checking%3A+Task+definition+and+dataset+construction%2E+Proceedings+of+the+ACL+2014+Workshop+on+Language+Technology+and+Computational+Social+Science+%2E) [our card](https://md.research.elamur.ai/free/en/papers/vlachos-2014-fact-checking-task-definition-dataset/) 4. Koby Crammer and Yoram Singer. 2001. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research 2(Dec):265–292. [find](https://scholar.google.com/scholar?q=Koby+Crammer+and+Yoram+Singer%2E+2001%2E+On+the+algorithmic+implementation+of+multiclass+kernel%2Dbased+vector+machines%2E+Journal+of+machine+learning+research+2%28Dec%29%3A265%E2%80%93292%2E) 5. William Yang Wang and Diyi Yang. 2015. That’s so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using \#petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (E… [find](https://scholar.google.com/scholar?q=William+Yang+Wang+and+Diyi+Yang%2E+2015%2E+That%E2%80%99s+so+annoying%21%21%21%3A+A+lexical+and+frame%2Dsemantic+embedding+based+data+augmentation+approach+to+automatic+categorization+of+annoying+behaviors+using+%5C%23petpeeve) [our card](https://md.research.elamur.ai/free/en/papers/wang-2015-that-s-so-annoying-lexical/) 6. Song Feng, Ritwik Banerjee, and Yejin Choi. 2012. Syntactic stylometry for deception detection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2. Association for Computational Linguistics, pages 171–175. [find](https://scholar.google.com/scholar?q=Song+Feng%2C+Ritwik+Banerjee%2C+and+Yejin+Choi%2E+2012%2E+Syntactic+stylometry+for+deception+detection%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Short+Papers) 7. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems. pages 649–657. [find](https://scholar.google.com/scholar?q=Xiang+Zhang%2C+Junbo+Zhao%2C+and+Yann+LeCun%2E+2015%2E+Character%2Dlevel+convolutional+networks+for+text+classification%2E+In+Advances+in+neural+information+processing+systems%2E+pages+649%E2%80%93657%2E) [our card](https://md.research.elamur.ai/free/en/papers/zhang-2015-character-level-convolutional-networks-text/) 8. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. ACL. [find](https://scholar.google.com/scholar?q=William+Ferreira+and+Andreas+Vlachos%2E+2016%2E+Emergent%3A+a+novel+data%2Dset+for+stance+classification%2E+In+Proceedings+of+the+2016+Conference+of+the+North+American+Chapter+of+the+Association+for+Computation) 9. Alex Graves and Jürgen Schmidhuber. 2005. Frame-wise phoneme classification with bidirectional lstm and other neural network architectures. Neural Networks 18(5):602–610. [find](https://scholar.google.com/scholar?q=Alex+Graves+and+Ju%CC%88rgen+Schmidhuber%2E+2005%2E+Frame%2Dwise+phoneme+classification+with+bidirectional+lstm+and+other+neural+network+architectures%2E+Neural+Networks+18%285%29%3A602%E2%80%93610%2E) 10. Zhen Hai, Peilin Zhao, Peng Cheng, Peng Yang, Xiao-Li Li, Guangxia Li, and Ant Financial. 2016. Deceptive review spam detection via exploiting task relatedness and unlabeled data. In EMNLP. [find](https://scholar.google.com/scholar?q=Zhen+Hai%2C+Peilin+Zhao%2C+Peng+Cheng%2C+Peng+Yang%2C+Xiao%2DLi+Li%2C+Guangxia+Li%2C+and+Ant+Financial%2E+2016%2E+Deceptive+review+spam+detection+via+exploiting+task+relatedness+and+unlabeled+data%2E+In+EMNLP%2E) 11. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780. [find](https://scholar.google.com/scholar?q=Sepp+Hochreiter+and+Ju%CC%88rgen+Schmidhuber%2E+1997%2E+Long+short%2Dterm+memory%2E+Neural+computation+9%288%29%3A1735%E2%80%931780%2E)