---
title: "Major Life Event Extraction from Twitter based on"
description: "The paper tackles a problem earlier Twitter event work had largely ignored: finding the… 0.62 precision. 38 references, cited by 26 papers here."
authors: ["Jiwei Li", "Alan Ritter", "Claire Cardie", "Eduard Hovy"]
year: 2014
venue: "Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)"
source: "https://aclanthology.org/D14-1214/"
doi: "10.3115/v1/d14-1214"
licence: "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)"
licence_url: "https://creativecommons.org/licenses/by-nc-sa/3.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2014-major-life-event-extraction-twitter/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2014-major-life-event-extraction-twitter/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/) / Major Life Event Extraction from Twitter based on Congratulations/Condolences Speech Acts

# Major Life Event Extraction from Twitter based on Congratulations/Condolences Speech Acts

**Authors:** [Jiwei Li](https://md.research.elamur.ai/free/en/authors/jiwei-li/), [Alan Ritter](https://md.research.elamur.ai/free/en/authors/alan-ritter/), [Claire Cardie](https://md.research.elamur.ai/free/en/authors/claire-cardie/) and [Eduard Hovy](https://md.research.elamur.ai/free/en/authors/eduard-hovy/) · **Year:** 2014 · **Venue:** [Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)](https://md.research.elamur.ai/free/en/venues/emnlp/) ([EMNLP 2014](https://md.research.elamur.ai/free/en/venues/emnlp/2014/)) · **Publisher:** [Association for Computational Linguistics](https://md.research.elamur.ai/free/en/publishers/association-for-computational-linguistics/) · **Licence:** [CC BY-NC-SA 3.0 (ACL Anthology, before 2016)](https://creativecommons.org/licenses/by-nc-sa/3.0/)

**Links:** [Publisher's page](https://aclanthology.org/D14-1214/) · [PDF](https://aclanthology.org/D14-1214.pdf) · [DOI 10.3115/v1/d14-1214](https://doi.org/10.3115/v1/d14-1214) · [OpenAlex](https://openalex.org/W2106191791)

**Citations:** 101 citations (OpenAlex, 2026-10-14) · [26](https://md.research.elamur.ai/free/en/papers/li-2014-major-life-event-extraction-twitter/cited-by/) in our library, of the 141,840 papers whose bibliographies we read

## Summary
The paper tackles a problem earlier Twitter event work had largely ignored: finding the private milestones of ordinary users, such as weddings, graduations, new jobs or births, and describing them in structured form. Such posts are rare, phrased in many ways, and mixed with wishes, gossip about others and past events, so labelled data is scarce. The authors avoid labelling the posts directly. Instead they look at replies: congratulations or condolences are expressed in a small set of phrasings and can be detected with high precision, and the message they answer is taken as a likely life event. Starting from seed replies, they cluster the answered conversations with LDA, label the coherent topics by hand, and bootstrap new reply patterns and topics. A three-stage pipeline then classifies the event type, checks that the speaker is personally involved, and extracts event properties with a CRF. On real-world identification the system reaches 0.62 precision and 0.48 recall, well above the compared approaches. The work shows that social reactions can serve as cheap supervision for rare, open-ended categories.

## Key points
- The data were drawn from CMU's 2011 Twitter Warehouse, a collection covering roughly 10% of all tweets posted that year. Each whole conversation, with the seed reply masked, was modelled as one LDA document (120 topics). Each sentence received a single topic, only verbs and nouns were kept, and inference used collapsed Gibbs sampling.
- Bootstrapping ran for 4 iterations and used streaming LDA to preserve topics already labelled. In each round, a person screened the top 20 replies to newly harvested tweets. The process ended with 30 congratulation/condolence patterns and 42 coherent life-event types.
- The pipeline has three stages:
  - Pipeline1 identifies the event category and filters out irrelevant tweets.
  - Pipeline2 keeps only self-reported events, so "my friend got married" is filtered out.
  - Pipeline3 extracts event properties, such as the spouse or the university, using a CRF.
- In end-to-end real-world identification, the system scored precision 0.62 and recall 0.48. The two compared approaches scored 0.13/0.20 and 0.25/0.18.
- Training the event classifier on topics from successive bootstrap steps (24, 38 and 42 classes) kept precision roughly flat (0.65, 0.64, 0.62). Recall rose steadily (0.36, 0.43, 0.48).

## Interesting ideas
- Label a hard-to-recognise message by the easy-to-recognise reaction it provokes; constrained reply speech acts act as a high-precision proxy annotator for a diverse, open-ended target class.
- Mask the trigger signal before clustering, so that topics reflect the content of the event rather than the seed words used to collect it.
- Restrict the topic model to verbs and nouns and assign one topic per sentence, which the authors found gives clearer, more coherent clusters than full bag-of-words.
- Break the task into a cascade that separates "is this an event?", "is the speaker involved?" and "what are its details?", with the involvement stage explicitly rejecting wishes, third parties and past mentions.
- Measure each bootstrap round separately to show that precision holds while recall grows, which justifies the iterations empirically.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): silver data for event or state-change frames could be harvested by finding texts followed by formulaic responses (thanks, apologies, congratulations) rather than labelling the events themselves.
- **I2** (learn): when building silver sets from seed patterns, removing the seed cue from features would act as a built-in guard against the model learning the harvesting trick instead of the phenomenon.
- **I3** (world): event-type induction over the world of a text could cluster predicate–argument heads from UD trees instead of raw tokens, keeping the clusters explainable.
- **I4** (frames): speech-act and frame analysis could carry explicit agent-is-speaker, modality (wish vs. fact) and tense checks before an event enters the text's world.
- **I5** (learn): self-induced repair rules or silver-data expansion rounds could be accepted only when per-round evaluation shows stable precision against a fixed held-out set.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [Stanford University](https://md.research.elamur.ai/free/en/universities/stanford-university/) — University, Stanford, United States · 310 works · 284,480 citations
- [Carnegie Mellon University](https://md.research.elamur.ai/free/en/universities/carnegie-mellon-university/) — University, Pittsburgh, United States · 365 works · 158,048 citations
- [Cornell University](https://md.research.elamur.ai/free/en/universities/cornell-university/) — University, Ithaca, United States · 89 works · 58,422 citations
- [The Ohio State University](https://md.research.elamur.ai/free/en/universities/the-ohio-state-university/) — University, Columbus, United States · 57 works · 17,793 citations

📚 Shelves (5): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 9,085 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 8,888 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 9,296 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 9,206 works · [🗄️ Information Systems](https://md.research.elamur.ai/free/en/shelves/computer-science/information-systems/) · 523 works

🧭 Topics (2): [🔎 Information extraction](https://md.research.elamur.ai/free/en/topics/information-extraction/) · 109 works · [💬 Speech act theory](https://md.research.elamur.ai/free/en/topics/speech-act-theory/) · 160 works

## Cite

Cite as: Jiwei Li, Alan Ritter, Claire Cardie and Eduard Hovy. 2014. Major Life Event Extraction from Twitter based on Congratulations/Condolences Speech Acts. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@inproceedings{li2014major,
  title = {{Major Life Event Extraction from Twitter based on Congratulations/Condolences Speech Acts}},
  author = {Jiwei Li, Alan Ritter, Claire Cardie and Eduard Hovy},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1214},
  url = {https://doi.org/10.3115/v1/d14-1214}
}
```

- [← previous A Comparison of the Events and Relations Across ACE, ERE, TAC-KBP, and FrameNet Annotation Standards](https://md.research.elamur.ai/free/en/papers/aguilar-2014-comparison-events-relations-across-ace/)
- [next → Cancelling the Maxim of Quantity: Another challenge for a Gricean theory of Scalar Implicatures](https://md.research.elamur.ai/free/en/papers/fox-2014-cancelling-maxim-quantity-another-challenge/)
- [↑ back to papers of 2014](https://md.research.elamur.ai/free/en/papers/2014/page/2/)

## References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hila Becker, Mor Naaman, and Luis Gravano. 2011. Beyond trending topics: Real-world event identification on twitter. ICWSM, 11:438–441. [find](https://scholar.google.com/scholar?q=Hila+Becker%2C+Mor+Naaman%2C+and+Luis+Gravano%2E+2011%2E+Beyond+trending+topics%3A+Real%2Dworld+event+identification+on+twitter%2E+ICWSM%2C+11%3A438%E2%80%93441%2E)
2. Cosmin Adrian Bejan, Matthew Titsworth, Andrew Hickl, and Sanda M Harabagiu. 2009. Nonparametric bayesian models for unsupervised event coreference resolution. In NIPS, pages 73–81. [find](https://scholar.google.com/scholar?q=Cosmin+Adrian+Bejan%2C+Matthew+Titsworth%2C+Andrew+Hickl%2C+and+Sanda+M+Harabagiu%2E+2009%2E+Nonparametric+bayesian+models+for+unsupervised+event+coreference+resolution%2E+In+NIPS%2C+pages+73%E2%80%9381%2E)
3. Edward Benson, Aria Haghighi, and Regina Barzilay. 2011. Event discovery in social media feeds. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 389–398. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Edward+Benson%2C+Aria+Haghighi%2C+and+Regina+Barzilay%2E+2011%2E+Event+discovery+in+social+media+feeds%2E+In+Proceedings+of+the+49th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Human+Langua)
4. David M Blei, Andrew Y Ng, and Michael I Jordan. 2003. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). the Journal of machine Learning research, 3:993–1022.
5. Morgane Ciot, Morgan Sonderegger, and Derek Ruths. 2013. Gender inference of twitter users in non-english contexts. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash, pages 18–21. [find](https://scholar.google.com/scholar?q=Morgane+Ciot%2C+Morgan+Sonderegger%2C+and+Derek+Ruths%2E+2013%2E+Gender+inference+of+twitter+users+in+non%2Denglish+contexts%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Proce)
6. Mark Craven, Johan Kumlien, et al. 1999. Constructing biological knowledge bases by extracting information from text sources. In ISMB, volume 1999, pages 77–86. [find](https://scholar.google.com/scholar?q=Mark+Craven%2C+Johan+Kumlien%2C+et+al%2E+1999%2E+Constructing+biological+knowledge+bases+by+extracting+information+from+text+sources%2E+In+ISMB%2C+volume+1999%2C+pages+77%E2%80%9386%2E)
7. Dmitry Davidov, Ari Rappoport, and Moshe Koppel. 2007. Fully unsupervised discovery of concept-specific relationships by web mining. In Annual Meeting-Association For Computational Linguistics, volume 45, page 232. [find](https://scholar.google.com/scholar?q=Dmitry+Davidov%2C+Ari+Rappoport%2C+and+Moshe+Koppel%2E+2007%2E+Fully+unsupervised+discovery+of+concept%2Dspecific+relationships+by+web+mining%2E+In+Annual+Meeting%2DAssociation+For+Computational+Linguistics%2C+volume)
8. Qiming Diao and Jing Jiang. 2013. A unified model for topics, events and users on twitter. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1869–1879. [find](https://scholar.google.com/scholar?q=Qiming+Diao+and+Jing+Jiang%2E+2013%2E+A+unified+model+for+topics%2C+events+and+users+on+twitter%2E+In+Proceedings+of+the+2013+Conference+on+Empirical+Methods+in+Natural+Language+Processing%2C+pages+1869%E2%80%931879%2E)
9. Qiming Diao, Jing Jiang, Feida Zhu, and Ee-Peng Lim. 2012. Finding bursty topics from microblogs. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pages 536–544. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Qiming+Diao%2C+Jing+Jiang%2C+Feida+Zhu%2C+and+Ee%2DPeng+Lim%2E+2012%2E+Finding+bursty+topics+from+microblogs%2E+In+Proceedings+of+the+50th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%3A+Long+Paper)
10. Thomas L Griffiths and Mark Steyvers. 2004. Finding scientific topics. Proceedings of the National academy of Sciences of the United States of America, 101(Suppl 1):5228–5235. [↗](https://doi.org/10.1073/pnas.0307752101)
11. Amit Gruber, Yair Weiss, and Michal Rosen-Zvi. 2007. Hidden topic markov models. In International Conference on Artificial Intelligence and Statistics, pages 163–170. [find](https://scholar.google.com/scholar?q=Amit+Gruber%2C+Yair+Weiss%2C+and+Michal+Rosen%2DZvi%2E+2007%2E+Hidden+topic+markov+models%2E+In+International+Conference+on+Artificial+Intelligence+and+Statistics%2C+pages+163%E2%80%93170%2E)
12. Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 541–550. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Raphael+Hoffmann%2C+Congle+Zhang%2C+Xiao+Ling%2C+Luke+Zettlemoyer%2C+and+Daniel+S+Weld%2E+2011%2E+Knowledge%2Dbased+weak+supervision+for+information+extraction+of+overlapping+relations%2E+In+Proceedings+of+the+49th+A)
13. Sean P Igo and Ellen Riloff. 2009. Corpus-based semantic lexicon induction with web-based corroboration. In Proceedings of the Workshop on Unsupervised and Minimally Supervised Learning of Lexical Semantics, pages 18–26. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Sean+P+Igo+and+Ellen+Riloff%2E+2009%2E+Corpus%2Dbased+semantic+lexicon+induction+with+web%2Dbased+corroboration%2E+In+Proceedings+of+the+Workshop+on+Unsupervised+and+Minimally+Supervised+Learning+of+Lexical+Sem)
14. Thorsten Joachims. 1999. Making large scale svm learning practical. [↗](https://doi.org/10.17877/de290r-14262)
15. Lingpeng Kong, Nathan Schneider, Swabha Swayamdipta, Archna Bhatia, Chris Dyer, and Noah Smith. 2014. [A dependency parser for tweets](https://md.research.elamur.ai/free/en/papers/kong-2014-dependency-parser-tweets/). In EMNLP.
16. Zornitsa Kozareva and Eduard Hovy. 2010a. Learning arguments and supertypes of semantic relations using recursive patterns. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1482–1491. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zornitsa+Kozareva+and+Eduard+Hovy%2E+2010a%2E+Learning+arguments+and+supertypes+of+semantic+relations+using+recursive+patterns%2E+In+Proceedings+of+the+48th+Annual+Meeting+of+the+Association+for+Computation)
17. Zornitsa Kozareva and Eduard Hovy. 2010b. Not all seeds are equal: Measuring the quality of text mining seeds. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 618–626. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zornitsa+Kozareva+and+Eduard+Hovy%2E+2010b%2E+Not+all+seeds+are+equal%3A+Measuring+the+quality+of+text+mining+seeds%2E+In+Human+Language+Technologies%3A+The+2010+Annual+Conference+of+the+North+American+Chapter+)
18. Zornitsa Kozareva, Ellen Riloff, and Eduard H Hovy. 2008. [Semantic class learning from the web with hyponym pattern linkage graphs](https://md.research.elamur.ai/free/en/papers/kozareva-2008-semantic-class-learning-web-hyponym/). In ACL, volume 8, pages 1048–1056.
19. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. [find](https://scholar.google.com/scholar?q=John+Lafferty%2C+Andrew+McCallum%2C+and+Fernando+CN+Pereira%2E+2001%2E+Conditional+random+fields%3A+Probabilistic+models+for+segmenting+and+labeling+sequence+data%2E)
20. Jiwei Li and Claire Cardie. 2013. Early stage influenza detection from twitter. arXiv preprint arXiv:1309.7340. [↗](https://arxiv.org/abs/1309.7340)
21. Jiwei Li and Claire Cardie. 2014. Timeline generation: Tracking individuals on twitter. [↗](https://doi.org/10.48550/arxiv.1309.7313)
22. Jiwei Li, Alan Ritter, and Eduard Hovy. 2014. Weakly supervised user profile extraction from twitter. ACL. [find](https://scholar.google.com/scholar?q=Jiwei+Li%2C+Alan+Ritter%2C+and+Eduard+Hovy%2E+2014%2E+Weakly+supervised+user+profile+extraction+from+twitter%2E+ACL%2E)
23. Yucheng Low, Deepak Agarwal, and Alexander J Smola. 2011. Multiple domain user personalization. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 123–131. ACM. [find](https://scholar.google.com/scholar?q=Yucheng+Low%2C+Deepak+Agarwal%2C+and+Alexander+J+Smola%2E+2011%2E+Multiple+domain+user+personalization%2E+In+Proceedings+of+the+17th+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+p)
24. David Mimno, Hanna M Wallach, Edmund Talley, Miriam Leenders, and Andrew McCallum. 2011. [Optimizing semantic coherence in topic models](https://md.research.elamur.ai/free/en/papers/mimno-2011-optimizing-semantic-coherence-topic-models/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 262– 272. Association for Computational Linguistics.
25. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2-Volume 2, pages 1003–1011. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mike+Mintz%2C+Steven+Bills%2C+Rion+Snow%2C+and+Dan+Jurafsky%2E+2009%2E+Distant+supervision+for+relation+extraction+without+labeled+data%2E+In+Proceedings+of+the+Joint+Conference+of+the+47th+Annual+Meeting+of+the+)
26. David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. 2010. [Automatic evaluation of topic coherence](https://md.research.elamur.ai/free/en/papers/newman-2010-automatic-evaluation-topic-coherence/). In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 100–108. Association for Computational Linguistics.
27. Olutobi Owoputi, Brendan O’Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. [Improved part-of-speech tagging for online conversational text with word clusters](https://md.research.elamur.ai/free/en/papers/owoputi-2013-improved-part-speech-tagging-online/). In Proceedings of NAACL-HLT, pages 380–390.
28. Ellen Riloff, Rosie Jones, et al. 1999. Learning dictionaries for information extraction by multi-level bootstrapping. In AAAI/IAAI, pages 474–479. [find](https://scholar.google.com/scholar?q=Ellen+Riloff%2C+Rosie+Jones%2C+et+al%2E+1999%2E+Learning+dictionaries+for+information+extraction+by+multi%2Dlevel+bootstrapping%2E+In+AAAI%2FIAAI%2C+pages+474%E2%80%93479%2E)
29. Alan Ritter, Colin Cherry, and Bill Dolan. 2010. [Unsupervised modeling of twitter conversations](https://md.research.elamur.ai/free/en/papers/ritter-2010-unsupervised-modeling-twitter-conversations/).
30. Alan Ritter, Sam Clark, Oren Etzioni, et al. 2011. [Named entity recognition in tweets: an experimental study](https://md.research.elamur.ai/free/en/papers/ritter-2011-named-entity-recognition-tweets-experimental/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1524–1534. Association for Computational Linguistics.
31. Alan Ritter, Oren Etzioni, Sam Clark, et al. 2012. Open domain event extraction from twitter. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1104–1112. ACM. [find](https://scholar.google.com/scholar?q=Alan+Ritter%2C+Oren+Etzioni%2C+Sam+Clark%2C+et+al%2E+2012%2E+Open+domain+event+extraction+from+twitter%2E+In+Proceedings+of+the+18th+ACM+SIGKDD+international+conference+on+Knowledge+discovery+and+data+mining%2C+pag)
32. Kirk Roberts and Sanda M Harabagiu. 2011. Unsupervised learning of selectional restrictions and detection of argument coercions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 980–990. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Kirk+Roberts+and+Sanda+M+Harabagiu%2E+2011%2E+Unsupervised+learning+of+selectional+restrictions+and+detection+of+argument+coercions%2E+In+Proceedings+of+the+Conference+on+Empirical+Methods+in+Natural+Langua)
33. Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010. Earthquake shakes twitter users: real-time event detection by social sensors. In Proceedings of the 19th international conference on World wide web, pages 851–860. ACM. [find](https://scholar.google.com/scholar?q=Takeshi+Sakaki%2C+Makoto+Okazaki%2C+and+Yutaka+Matsuo%2E+2010%2E+Earthquake+shakes+twitter+users%3A+real%2Dtime+event+detection+by+social+sensors%2E+In+Proceedings+of+the+19th+international+conference+on+World+wide)
34. Roser Saurı́ and James Pustejovsky. 2007. Determining modality and factuality for text entailment. In Semantic Computing, 2007. ICSC 2007. International Conference on, pages 509–516. IEEE. [find](https://scholar.google.com/scholar?q=Roser+Saur%C4%B1%CC%81+and+James+Pustejovsky%2E+2007%2E+Determining+modality+and+factuality+for+text+entailment%2E+In+Semantic+Computing%2C+2007%2E+ICSC+2007%2E+International+Conference+on%2C+pages+509%E2%80%93516%2E+IEEE%2E)
35. Roser Saurı́ and James Pustejovsky. 2009. Factbank: A corpus annotated with event factuality. Language resources and evaluation, 43(3):227–268. [find](https://scholar.google.com/scholar?q=Roser+Saur%C4%B1%CC%81+and+James+Pustejovsky%2E+2009%2E+Factbank%3A+A+corpus+annotated+with+event+factuality%2E+Language+resources+and+evaluation%2C+43%283%29%3A227%E2%80%93268%2E)
36. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y Ng. 2008. [Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks](https://md.research.elamur.ai/free/en/papers/snow-2008-cheap-fast-but-is-it/). In Proceedings of the conference on empirical methods in natural language processing, pages 254–263. Association for Computational Linguistics.
37. Limin Yao, David Mimno, and Andrew McCallum. 2009. Efficient methods for topic model inference on streaming document collections. [find](https://scholar.google.com/scholar?q=Limin+Yao%2C+David+Mimno%2C+and+Andrew+McCallum%2E+2009%2E+Efficient+methods+for+topic+model+inference+on+streaming+document+collections%2E)
38. Limin Yao, Aria Haghighi, Sebastian Riedel, and Andrew McCallum. 2011. [Structured relation discovery using generative models](https://md.research.elamur.ai/free/en/papers/yao-2011-structured-relation-discovery-generative-models/). In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1456–1466. Association for Computational Linguistics.

*Updated 24 October 2026*