---
title: "Emergent: a novel data-set for stance classification — Ferreira 2016"
description: "The authors present Emergent, built from a database dump of a rumour-debunking project run with the… 73% accuracy. 23 references, cited by 108 papers here."
authors: ["William Ferreira", "Andreas Vlachos"]
year: 2016
venue: "Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies"
source: "https://aclanthology.org/N16-1138/"
doi: "10.18653/v1/n16-1138"
licence: "CC BY 4.0 (ACL Anthology)"
licence_url: "https://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/ferreira-2016-emergent-novel-data-set-stance/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/ferreira-2016-emergent-novel-data-set-stance/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2016](https://md.research.elamur.ai/free/en/papers/2016/) / Emergent: a novel data-set for stance classification

# Emergent: a novel data-set for stance classification

**Authors:** William Ferreira and [Andreas Vlachos](https://md.research.elamur.ai/free/en/authors/andreas-vlachos/) · **Year:** 2016 · **Venue:** Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · **Licence:** [CC BY 4.0 (ACL Anthology)](https://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://aclanthology.org/N16-1138/) · [PDF](https://aclanthology.org/N16-1138.pdf) · [DOI 10.18653/v1/n16-1138](https://doi.org/10.18653/v1/n16-1138) · [OpenAlex](https://openalex.org/W2337875011)

**Citations:** 348 citations (OpenAlex, 2026-10-14) · [108](https://md.research.elamur.ai/free/en/papers/ferreira-2016-emergent-novel-data-set-stance/cited-by/) in our library, of the 140,884 papers whose bibliographies we read

## Summary
Automated fact-checking needs realistic supervised data. Earlier resources such as Politifact-based sets give only a truthfulness rating per claim. Earlier stance studies used very few claims or let the same claims appear in both training and testing. The authors present Emergent, built from a database dump of a rumour-debunking project run with the Tow Center for Digital Journalism. It holds 300 claims, each with a veracity verdict (true, false, unverified), and 2,595 linked news articles. Journalists condensed each article into a headline and tagged its stance as for, against or observing. Because these labels came from the journalists' normal work, the data is real-world and cost nothing extra to annotate. The paper then classifies headline stance toward a claim using L1-regularized multiclass logistic regression. Its features look at the headline alone and at how the headline relates to the claim, using syntax, PPDB alignment and SVO matching. On a test set with no shared claims, the model reaches 73% accuracy. This is well above a lexical-overlap baseline (32%) and the Excitement Open Platform retrained on the same data (47%).

## Key points
- The dataset has 300 claims and 2,595 headlines, averaging 8.65 articles per claim (range 1–50). Stance labels are distributed as 47.7% for, 15.2% against and 37.1% observing.
- The train/test split is 2,071/524 instances, with each claim placed in only one part. Data and code are released at github.com/willferreira/mscproject.
- The features are:
  - **BoW** and a question-mark flag (**Q**);
  - **RootDist**: the dependency-tree distance from the root to refuting or hedging words, using Stanford CoreNLP parses;
  - a **PPDB** + Kuhn-Munkres word-alignment score;
  - a negation flag on aligned words;
  - **SVO** triple relations from PPDB;
  - **word2vec** cosine similarity, with vectors composed by multiplication.
- Results on the test set:
  - The proposed model reaches 73% accuracy.
  - A majority baseline gets 47% and an overlap-threshold baseline gets 32%.
  - EOP (MaxEntClassificationEDA) gets 33% with its RTE-3 pre-trained model and 47% when retrained on Emergent.
- Ablation results:
  - Removing BoW causes the largest test drop (5.15%), followed by RootDist (2.48%).
  - Restricting the features to just BoW, Q and word2vec yields a score 3% below the full model.
  - word2vec contributed nothing on test, which the authors partly attribute to context-based embeddings failing to capture antonymy.

## Interesting ideas
- **Harvest labels from professional workflows.** Journalists' routine verification produced stance, veracity and summary labels with no extra annotation effort. Newsrooms or fact-checking teams working in other languages, including Ukrainian, could be a similar source of free, realistic labels.
- **Split by claim, not by instance.** Keeping every claim entirely in either training or test measures how well a model handles unseen claims. The same design applies to any paired-text task where topics repeat, such as entailment, QA or retrieval.
- **Measure syntactic distance to cue words.** Counting dependency edges from the root to hedging or denial words separates mere reporting from endorsement better than simply detecting those words. This cheap feature could carry over to hedge detection, sentiment or claim detection in any language with a parser.
- **Turn paraphrase alignment into a single score.** Each claim–headline word pair is scored by stem match, PPDB paraphrase or a penalty. The Hungarian algorithm then finds the best one-to-one matching, and its normalized score becomes one feature. This gives a transparent similarity signal for low-resource settings where only a paraphrase lexicon is available.
- **Use L1 sparsity for interpretation.** L1 regularization drives many weights to zero, so the surviving non-zero weights show which features each class relies on. This makes ablation results easier to explain for any feature-engineered classifier.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

Fields: [Natural language processing](https://md.research.elamur.ai/free/en/shelves/nlp/)

## Institutions

- [University of Sheffield](https://md.research.elamur.ai/free/en/universities/university-of-sheffield/) — University, Sheffield, United Kingdom · 67 works · 17,924 citations
- [University College London](https://md.research.elamur.ai/free/en/universities/university-college-london/) — University, London, United Kingdom · 44 works · 14,853 citations

## Cite

Cite as: William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{ferreira2016emergent,
  title = {{Emergent: a novel data-set for stance classification}},
  author = {William Ferreira and Andreas Vlachos},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1138},
  url = {https://doi.org/10.18653/v1/n16-1138}
}
```

- [← previous A Persona-Based Neural Conversation Model](https://md.research.elamur.ai/free/en/papers/li-2016-persona-based-neural-conversation-model/)
- [next → Neural versus Phrase-Based Machine Translation Quality: a Case Study](https://md.research.elamur.ai/free/en/papers/bentivogli-2016-neural-versus-phrase-based-machine/)
- [↑ back to papers of 2016](https://md.research.elamur.ai/free/en/papers/2016/)

## References

23 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Samuel+R%2E+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D%2E+Manning%2E+2015%2E+A+large+annotated+corpus+for+learning+natural+language+inference%2E+In+Proceedings+of+the+2015+Conference+on+Empirica)
2. Sarah Cohen, Chengkai Li, Jun Yang, and Cong Yu. 2011. Computational Journalism: a call to arms to database researchers. Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR 2011) Asilomar, California, USA., pages 148– 151. [find](https://scholar.google.com/scholar?q=Sarah+Cohen%2C+Chengkai+Li%2C+Jun+Yang%2C+and+Cong+Yu%2E+2011%2E+Computational+Journalism%3A+a+call+to+arms+to+database+researchers%2E+Proceedings+of+the+5th+Biennial+Conference+on+Innovative+Data+Systems+Research+)
3. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Josef Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, Steve Pulman, Ted Briscoe, Holger Maier, and Karsten Konrad. 1996. Using the framework. Technical report, The FraCas Consortium. [find](https://scholar.google.com/scholar?q=Robin+Cooper%2C+Dick+Crouch%2C+Jan+Van+Eijck%2C+Chris+Fox%2C+Josef+Van+Genabith%2C+Jan+Jaspars%2C+Hans+Kamp%2C+David+Milward%2C+Manfred+Pinkal%2C+Massimo+Poesio%2C+Steve+Pulman%2C+Ted+Briscoe%2C+Holger+Maier%2C+and+Karsten+Kon)
4. Kazi Saidul Hasan and Vincent Ng. 2013. Stance Classification of Ideological Debates: Data, Models, Features, and Constraints. In IJCNLP, pages 1348–1356. Asian Federation of Natural Language Processing / ACL. [find](https://scholar.google.com/scholar?q=Kazi+Saidul+Hasan+and+Vincent+Ng%2E+2013%2E+Stance+Classification+of+Ideological+Debates%3A+Data%2C+Models%2C+Features%2C+and+Constraints%2E+In+IJCNLP%2C+pages+1348%E2%80%931356%2E+Asian+Federation+of+Natural+Language+Processi)
5. Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015. Detecting check-worthy factual claims in presidential debates. In Proceedings of the 24th ACM International Conference on Information and Knowledge Management. [find](https://scholar.google.com/scholar?q=Naeemul+Hassan%2C+Chengkai+Li%2C+and+Mark+Tremayne%2E+2015%2E+Detecting+check%2Dworthy+factual+claims+in+presidential+debates%2E+In+Proceedings+of+the+24th+ACM+International+Conference+on+Information+and+Knowledg)
6. Harold W. Kuhn. 1955. The Hungarian Method for the assignment problem. Naval Research Logistics Quarterly, 2:83–97. [find](https://scholar.google.com/scholar?q=Harold+W%2E+Kuhn%2E+1955%2E+The+Hungarian+Method+for+the+assignment+problem%2E+Naval+Research+Logistics+Quarterly%2C+2%3A83%E2%80%9397%2E)
7. Xiaomo Liu, Armineh Nourbakhsh, Quanzhi Li, Rui Fang, and Sameena Shah. 2015. Real-time rumor debunking on twitter. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pages 1867–1870. [find](https://scholar.google.com/scholar?q=Xiaomo+Liu%2C+Armineh+Nourbakhsh%2C+Quanzhi+Li%2C+Rui+Fang%2C+and+Sameena+Shah%2E+2015%2E+Real%2Dtime+rumor+debunking+on+twitter%2E+In+Proceedings+of+the+24th+ACM+International+on+Conference+on+Information+and+Knowle)
8. Michael Lukasik, Trevor Cohn, and Kalina Bontcheva. 2015. Classifying tweet level judgements of rumours in social media. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2590–2595, Lisbon, Portugal, September. Association for Computational Linguistics.… [find](https://scholar.google.com/scholar?q=Michael+Lukasik%2C+Trevor+Cohn%2C+and+Kalina+Bontcheva%2E+2015%2E+Classifying+tweet+level+judgements+of+rumours+in+social+media%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Language+)
9. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60. [our card](https://md.research.elamur.ai/free/en/papers/manning-2014-stanford-corenlp-natural-language-processing/)
10. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), Reykja… [our card](https://md.research.elamur.ai/free/en/papers/marelli-2014-sick-cure-evaluation-compositional-distributional/)
11. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-efficient-estimation-word-representations-vector/)
12. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, p… [our card](https://md.research.elamur.ai/free/en/papers/mikolov-2013-distributed-representations-words-phrases-their/)
13. Curran Associates, Inc. James Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the Society for Industrial and Applied Mathematics, 5(1):32–38. [find](https://scholar.google.com/scholar?q=Curran+Associates%2C+Inc%2E+James+Munkres%2E+1957%2E+Algorithms+for+the+assignment+and+transportation+problems%2E+Journal+of+the+Society+for+Industrial+and+Applied+Mathematics%2C+5%281%29%3A32%E2%80%9338%2E)
14. Ndapandula Nakashole and Tom M Mitchell. 2014. Language-Aware Truth Assessment of Fact Candidates. Acl, pages 1009–1019. [find](https://scholar.google.com/scholar?q=Ndapandula+Nakashole+and+Tom+M+Mitchell%2E+2014%2E+Language%2DAware+Truth+Assessment+of+Fact+Candidates%2E+Acl%2C+pages+1009%E2%80%931019%2E)
15. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. Ppdb 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational… [our card](https://md.research.elamur.ai/free/en/papers/pavlick-2015-ppdb-2-0-better-paraphrase/)
16. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:282… [find](https://scholar.google.com/scholar?q=F%2E+Pedregosa%2C+G%2E+Varoquaux%2C+A%2E+Gramfort%2C+V%2E+Michel%2C+B%2E+Thirion%2C+O%2E+Grisel%2C+M%2E+Blondel%2C+P%2E+Prettenhofer%2C+R%2E+Weiss%2C+V%2E+Dubourg%2C+J%2E+Vanderplas%2C+A%2E+Passos%2C+D%2E+Cournapeau%2C+M%2E+Brucher%2C+M%2E+Perrot%2C+and+E%2E+Duc)
17. Vahed Qazvinian, Emily Rosengren, Dragomir R. Radev, and Qiaozhu Mei. 2011. Rumor has it: Identifying misinformation in microblogs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1589–1599. [our card](https://md.research.elamur.ai/free/en/papers/qazvinian-2011-rumor-has-it-identifying-misinformation/)
18. Vasile Rus and Mihai Lintean. 2012. A comparison of greedy and optimal assessment of natural language student input using word-to-word similarity metrics. In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, pages 157–162, Stroudsburg, PA, USA. Association for Compu… [find](https://scholar.google.com/scholar?q=Vasile+Rus+and+Mihai+Lintean%2E+2012%2E+A+comparison+of+greedy+and+optimal+assessment+of+natural+language+student+input+using+word%2Dto%2Dword+similarity+metrics%2E+In+Proceedings+of+the+Seventh+Workshop+on+Bui)
19. Craig Silverman. 2015. Lies, Damn Lies and Viral Content. [http://towcenter.org/research/](http://towcenter.org/research/) lies-damn-lies-and-viral-content/, February. [↗](http://towcenter.org/research/)
20. Andreas Vlachos and Sebastian Riedel. 2014. Fact Checking: Task definition and dataset construction. Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22. [our card](https://md.research.elamur.ai/free/en/papers/vlachos-2014-fact-checking-task-definition-dataset/)
21. Andreas Vlachos and Sebastian Riedel. 2015. Identification and verification of simple claims about statistical properties. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. [find](https://scholar.google.com/scholar?q=Andreas+Vlachos+and+Sebastian+Riedel%2E+2015%2E+Identification+and+verification+of+simple+claims+about+statistical+properties%2E+In+Proceedings+of+the+2015+Conference+on+Empirical+Methods+in+Natural+Languag)
22. Marilyn Walker, Pranav Anand, Rob Abbott, and Ricky Grant. 2012. Stance Classification using Dialogic Properties of Persuasion. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 592–596. [our card](https://md.research.elamur.ai/free/en/papers/walker-2012-stance-classification-dialogic-properties-persuasion/)
23. Zhe Zhao, Paul Resnick, and Qiaozhu Mei. 2015. Enquiring minds: Early detection of rumors in social media from enquiry posts. In Proceedings of the 24th International Conference on World Wide Web, pages 1395–1405. [find](https://scholar.google.com/scholar?q=Zhe+Zhao%2C+Paul+Resnick%2C+and+Qiaozhu+Mei%2E+2015%2E+Enquiring+minds%3A+Early+detection+of+rumors+in+social+media+from+enquiry+posts%2E+In+Proceedings+of+the+24th+International+Conference+on+World+Wide+Web%2C+pag)