---
title: "Goal Driven Discovery of Distributional Differences via Language"
description: "Open-ended discovery has no reference answers, so the authors build two evaluation resources. 91 references, cited by 38 papers here."
authors: ["Ruiqi Zhong", "Peter Zhang", "Steve Li", "Jinwoo Ahn", "Dan Klein", "Jacob Steinhardt"]
year: 2023
venue: "arXiv:2302.14233"
source: "https://arxiv.org/abs/2302.14233"
doi: "10.48550/arxiv.2302.14233"
arxiv: "2302.14233"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/zhong-2023-goal-driven-discovery-distributional-differences/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/zhong-2023-goal-driven-discovery-distributional-differences/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/) / Goal Driven Discovery of Distributional Differences via Language Descriptions

# Goal Driven Discovery of Distributional Differences via Language Descriptions

**Authors:** [Ruiqi Zhong](https://md.research.elamur.ai/free/en/authors/ruiqi-zhong/), Peter Zhang, Steve Li, [Jinwoo Ahn](https://md.research.elamur.ai/free/en/authors/jinwoo-ahn/), [Dan Klein](https://md.research.elamur.ai/free/en/authors/dan-klein/) and [Jacob Steinhardt](https://md.research.elamur.ai/free/en/authors/jacob-steinhardt/) · **Year:** 2023 · **Venue:** arXiv:2302.14233 · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2302.14233) · [PDF](https://arxiv.org/pdf/2302.14233) · [DOI 10.48550/arxiv.2302.14233](https://doi.org/10.48550/arxiv.2302.14233) · [arXiv 2302.14233](https://arxiv.org/abs/2302.14233) · [OpenAlex](https://openalex.org/W4322759712)

**Citations:** 4 citations (OpenAlex, 2026-10-14) · [38](https://md.research.elamur.ai/free/en/papers/zhong-2023-goal-driven-discovery-distributional-differences/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Comparing two large text collections by hand is slow and ad hoc. The authors frame this as an ML task called D5. A system receives a user's exploration goal and a pair of corpora. It must return natural-language predicates that describe how the corpora differ in ways that matter for that goal. For example, reports from patients on one drug might more often mention paranoia. Outputs are judged on two criteria: validity (the difference really holds) and relevance to the goal.

Open-ended discovery has no reference answers, so the authors build two evaluation resources. SynD5 contains synthetic corpus pairs with a known goal-relevant difference plus a more salient distractor, and it is scored automatically by an LM judge. OpenD5 collects 675 real problems from many fields, and its outputs are partly assessed by humans. A goal-conditioned version of an earlier system (Zhong et al., 2022) outperforms the goal-agnostic one on both benchmarks. It also surfaces findings the authors did not know before.

This work makes exploratory text analysis measurable and learnable. However, the discoveries reflect correlation, not causation, and could reinforce biases if misused.

## Key points
- **OpenD5** contains 675 problems drawn from domains including health, machine learning, the humanities, the social sciences and business.
  - It totals about 4.4M text samples, with roughly 17K per corpus pair on average.
  - Each pair is split into an exploration half and a validation half.
- **SynD5** has 300 synthetic problems.
  - Corpora vary along two of three dimensions: topic, genre and language. One dimension is goal-relevant and the other is a distractor.
  - The share of samples showing the goal-relevant difference ranges from 0.6 to 1.
  - Claude-v1.3, prompted with few-shot examples, checks whether an output matches the reference.
- **Effect of conditioning on the goal:**
  - 12% more correct answers on SynD5.
  - Relevant candidate discoveries 31% more often on OpenD5.
- **Example findings** include insights from commercial reviews, demographic and temporal differences in discussion topics, political stances in speeches, lyric styles, and NLP model error patterns.
- **Self-supervised training:** a proof-of-concept trains LMs on unlabeled problems to propose more valid hypotheses.
- **Released resources:** code is at github.com/ruiqi-zhong/D5, and the OpenD5 download scripts are at github.com/petezh/OpenD5.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Berkeley](https://md.research.elamur.ai/free/en/universities/university-of-california-berkeley/) — University, Berkeley, United States · 209 works · 244,782 citations

📚 Shelves (6): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works · [🤝 Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/) · 1,954 works · [🤝 General Social Sciences](https://md.research.elamur.ai/free/en/shelves/social-sciences/general-social-sciences/) · 254 works

## Cite

Cite as: Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein and Jacob Steinhardt. 2023. Goal Driven Discovery of Distributional Differences via Language Descriptions. arXiv:2302.14233.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{zhong2023goal,
  title = {{Goal Driven Discovery of Distributional Differences via Language Descriptions}},
  author = {Ruiqi Zhong, Peter Zhang, Steve Li, Jinwoo Ahn, Dan Klein and Jacob Steinhardt},
  year = {2023},
  doi = {10.48550/arxiv.2302.14233},
  eprint = {2302.14233},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.14233}
}
```

- [← previous Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages](https://md.research.elamur.ai/free/en/papers/hu-2023-large-multilingual-models-pivot-zero/)
- [next → Paloma: A Benchmark for Evaluating Language Model Fit](https://md.research.elamur.ai/free/en/papers/magnusson-2023-paloma-benchmark-evaluating-language-model/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/11/)

## References

91 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. The hewlett foundation: Automated essay scoring, 2012. competitions/asap-aes. [find](https://scholar.google.com/scholar?q=The+hewlett+foundation%3A+Automated+essay+scoring%2C+2012%2E+competitions%2Fasap%2Daes%2E)
2. URL [https://kaggle.com/The](https://kaggle.com/The) hewlett foundation: Short answer scoring, 2013. URL https://kaggle.com/competitions/ asap-sas. [↗](https://kaggle.com/The)
3. Ad observer. [https://adobserver.org/](https://adobserver.org/), 2021. Accessed: 2022-12-30. [↗](https://adobserver.org/)
4. Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. [Scaling laws for generative mixed-modal language models](https://md.research.elamur.ai/free/en/papers/aghajanyan-2023-scaling-laws-generative-mixed-modal/). arXiv preprint arXiv:2301.03728, 2023.
5. Roee Aharoni and Yoav Goldberg. [Unsupervised domain clusters in pretrained language models](https://md.research.elamur.ai/free/en/papers/aharoni-2020-unsupervised-domain-clusters-pretrained-language/). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7747–7763, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. acl-main.692. UR…
6. Mohammad Alali, Shaayan Syed, Mohammed Alsayed, Smit Patel, and Hemanth Bodala. Justice: A benchmark dataset for supreme court’s judgment prediction. arXiv preprint arXiv:2112.03414, 2021. [find](https://scholar.google.com/scholar?q=Mohammad+Alali%2C+Shaayan+Syed%2C+Mohammed+Alsayed%2C+Smit+Patel%2C+and+Hemanth+Bodala%2E+Justice%3A+A+benchmark+dataset+for+supreme+court%E2%80%99s+judgment+prediction%2E+arXiv+preprint+arXiv%3A2112%2E03414%2C+2021%2E)
7. Akari Asai, Sara Evensen, Behzad Golshan, Alon Halevy, Vivian Li, Andrei Lopatenko, Daniela Stepanov, Yoshihiko Suhara, Wang-Chiew Tan, and Yinzhan Xu. Happydb: A corpus of 100,000 crowdsourced happy moments. arXiv preprint arXiv:1801.07746, 2018. [find](https://scholar.google.com/scholar?q=Akari+Asai%2C+Sara+Evensen%2C+Behzad+Golshan%2C+Alon+Halevy%2C+Vivian+Li%2C+Andrei+Lopatenko%2C+Daniela+Stepanov%2C+Yoshihiko+Suhara%2C+Wang%2DChiew+Tan%2C+and+Yinzhan+Xu%2E+Happydb%3A+A+corpus+of+100%2C000+crowdsourced+happy+)
8. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. [find](https://scholar.google.com/scholar?q=Yuntao+Bai%2C+Andy+Jones%2C+Kamal+Ndousse%2C+Amanda+Askell%2C+Anna+Chen%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Stanislav+Fort%2C+Deep+Ganguli%2C+Tom+Henighan%2C+et+al%2E+Training+a+helpful+and+harmless+assistant+with+reinforcem)
9. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. [Constitutional ai: Harmlessness from ai feedback](https://md.research.elamur.ai/free/en/papers/bai-2022-constitutional-ai-harmlessness-ai-feedback/). arXiv preprint arXiv:2212.08073, 2022b.
10. Miriam Barnum and James Lo. Is the npt unraveling? evidence from text analysis of review conference statements. Journal of Peace Research, 57(6):740–751, 2020. [find](https://scholar.google.com/scholar?q=Miriam+Barnum+and+James+Lo%2E+Is+the+npt+unraveling%3F+evidence+from+text+analysis+of+review+conference+statements%2E+Journal+of+Peace+Research%2C+57%286%29%3A740%E2%80%93751%2C+2020%2E)
11. Alexander Baturo, Niheer Dasandi, and Slava J Mikhaylov. Understanding state preferences with text as data: Introducing the un general debate corpus. Research & Politics, 4(2):2053168017712821, 2017. [find](https://scholar.google.com/scholar?q=Alexander+Baturo%2C+Niheer+Dasandi%2C+and+Slava+J+Mikhaylov%2E+Understanding+state+preferences+with+text+as+data%3A+Introducing+the+un+general+debate+corpus%2E+Research+%26+Politics%2C+4%282%29%3A2053168017712821%2C+2017%2E)
12. Akshay Bhalotia. Yc company scraper. [https://github.com/akshaybhalotia/yc_company_](https://github.com/akshaybhalotia/yc_company_) scraper, 2022. [↗](https://github.com/akshaybhalotia/yc_company_)
13. Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.", 2009. [find](https://scholar.google.com/scholar?q=Steven+Bird%2C+Ewan+Klein%2C+and+Edward+Loper%2E+Natural+language+processing+with+Python%3A+analyzing+text+with+the+natural+language+toolkit%2E+%22+O%E2%80%99Reilly+Media%2C+Inc%2E%22%2C+2009%2E)
14. David M Blei, Andrew Y Ng, and Michael I Jordan. [Latent dirichlet allocation](https://md.research.elamur.ai/free/en/papers/blei-2003-latent-dirichlet-allocation/). Journal of machine Learning research, 3(Jan):993–1022, 2003.
15. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. The snli corpus. 2015. [find](https://scholar.google.com/scholar?q=Samuel+R+Bowman%2C+Gabor+Angeli%2C+Christopher+Potts%2C+and+Christopher+D+Manning%2E+The+snli+corpus%2E+2015%2E)
16. Divy Bramhecha. Poetry Foundation Poems, 2019. URL [https://www.kaggle.com/datasets/](https://www.kaggle.com/datasets/) tgdivy/poetry-foundation-poems. [↗](https://www.kaggle.com/datasets/)
17. Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, and Dan Jurafsky. Replication code and data for “Computational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration” [dataset]. https:… [find](https://scholar.google.com/scholar?q=Dallas+Card%2C+Serina+Chang%2C+Chris+Becker%2C+Julia+Mendelsohn%2C+Rob+Voigt%2C+Leah+Boustan%2C+Ran+Abramitzky%2C+and+Dan+Jurafsky%2E+Replication+code+and+data+for+%E2%80%9CComputational+analysis+of+140+years+of+US+political)
18. Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. [Neural legal judgment prediction in english](https://md.research.elamur.ai/free/en/papers/chalkidis-2019-neural-legal-judgment-prediction-english/). arXiv preprint arXiv:1906.02059, 2019.
19. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims. In Proc. of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. [URL http:…](https://md.research.elamur.ai/free/en/papers/chen-2019-seeing-things-different-angle-discovering/)
20. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. [find](https://scholar.google.com/scholar?q=Hyung+Won+Chung%2C+Le+Hou%2C+Shayne+Longpre%2C+Barret+Zoph%2C+Yi+Tay%2C+William+Fedus%2C+Eric+Li%2C+Xuezhi+Wang%2C+Mostafa+Dehghani%2C+Siddhartha+Brahma%2C+et+al%2E+Scaling+instruction%2Dfinetuned+language+models%2E+arXiv+prep)
21. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. [Boolq: Exploring the surprising difficulty of natural yes/no questions](https://md.research.elamur.ai/free/en/papers/clark-2019-boolq-exploring-surprising-difficulty-natural/). arXiv preprint arXiv:1905.10044, 2019.
22. Norman R Draper and Harry Smith. Applied regression analysis, volume 326. John Wiley & Sons, 1998. [find](https://scholar.google.com/scholar?q=Norman+R+Draper+and+Harry+Smith%2E+Applied+regression+analysis%2C+volume+326%2E+John+Wiley+%26+Sons%2C+1998%2E)
23. Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christopher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross-modal embeddings. arXiv preprint arXiv:2203.14960, 2022. — [arXiv:2203.14960](https://arxiv.org/abs/2203.14960) [find](https://scholar.google.com/scholar?q=Sabri+Eyuboglu%2C+Maya+Varma%2C+Khaled+Saab%2C+Jean%2DBenoit+Delbrouck%2C+Christopher+Lee%2DMesser%2C+Jared+Dunnmon%2C+James+Zou%2C+and+Christopher+R%C3%A9%2E+Domino%3A+Discovering+systematic+errors+with+cross%2Dmodal+embeddings%2E)
24. Yujia Gao, Jinu Jang, and Diyi Yang. Understanding the usage of online media for parenting from infancy to preschool at scale. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–12, 2021. [find](https://scholar.google.com/scholar?q=Yujia+Gao%2C+Jinu+Jang%2C+and+Diyi+Yang%2E+Understanding+the+usage+of+online+media+for+parenting+from+infancy+to+preschool+at+scale%2E+In+Proceedings+of+the+2021+CHI+Conference+on+Human+Factors+in+Computing+S)
25. Anthony G Greenwald and Mahzarin R Banaji. Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological review, 102(1):4, 1995. [find](https://scholar.google.com/scholar?q=Anthony+G+Greenwald+and+Mahzarin+R+Banaji%2E+Implicit+social+cognition%3A+attitudes%2C+self%2Desteem%2C+and+stereotypes%2E+Psychological+review%2C+102%281%29%3A4%2C+1995%2E)
26. Ivan Habernal and Iryna Gurevych. [Which argument is more convincing? Analyzing and predicting convincingness of Web arguments using bidirectional LSTM](https://md.research.elamur.ai/free/en/papers/habernal-2016-which-argument-is-more-convincing/). In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1589–1599, Berlin, Germany,…
27. Kevin Hartman. Advertisement Transcripts from Various Industries, 2019. URL [https://tinyurl.com/5w36dwdx](https://tinyurl.com/5w36dwdx). [↗](https://tinyurl.com/5w36dwdx)
28. He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. [Decoupling strategy and generation in negotiation dialogues](https://md.research.elamur.ai/free/en/papers/he-2018-decoupling-strategy-generation-negotiation-dialogues/), 2018.
29. Jibo He. Big Data Set from RateMyProfessor.com for Professors’ Teaching Evaluation, 2020. URL [https://data.mendeley.com/datasets/fvtfjyvw7d/2](https://data.mendeley.com/datasets/fvtfjyvw7d/2). [↗](https://data.mendeley.com/datasets/fvtfjyvw7d/2)
30. Samuel He. Goodbye world: using natural language processing to identify suicidal posts, 2021. URL [https://github.com/hesamuel/goodbye_world](https://github.com/hesamuel/goodbye_world). [↗](https://github.com/hesamuel/goodbye_world)
31. James J Heckman and Burton Singer. Abducting economics. American Economic Review, 107(5): 298–302, 2017. [find](https://scholar.google.com/scholar?q=James+J+Heckman+and+Burton+Singer%2E+Abducting+economics%2E+American+Economic+Review%2C+107%285%29%3A+298%E2%80%93302%2C+2017%2E)
32. Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2021. [find](https://scholar.google.com/scholar?q=Evan+Hernandez%2C+Sarah+Schwettmann%2C+David+Bau%2C+Teona+Bagashvili%2C+Antonio+Torralba%2C+and+Jacob+Andreas%2E+Natural+language+descriptions+of+deep+visual+features%2E+In+International+Conference+on+Learning+Repr)
33. Or Honovich, Uri Shaham, Samuel R Bowman, and Omer Levy. [Instruction induction: From few examples to natural language task descriptions](https://md.research.elamur.ai/free/en/papers/honovich-2022-instruction-induction-few-examples-natural/). arXiv preprint arXiv:2205.10782, 2022.
34. Nabil Hossain, John Krumm, and Michael Gamon. " president vows to cut\\< taxes\> hair": Dataset and analysis of creative text editing for humorous headlines. arXiv preprint arXiv:1906.00274, 2019. — [arXiv:1906.00274](https://arxiv.org/abs/1906.00274) [find](https://scholar.google.com/scholar?q=Nabil+Hossain%2C+John+Krumm%2C+and+Michael+Gamon%2E+%22+president+vows+to+cut%5C%3C+taxes%5C%3E+hair%22%3A+Dataset+and+analysis+of+creative+text+editing+for+humorous+headlines%2E+arXiv+preprint+arXiv%3A1906%2E00274%2C+2019%2E+%E2%80%94+%5Ba)
35. Kaggle. TMDB 5000 Movie Dataset, 2018. URL [https://www.kaggle.com/datasets/tmdb/](https://www.kaggle.com/datasets/tmdb/) tmdb-movie-metadata. [↗](https://www.kaggle.com/datasets/tmdb/)
36. Rohit Kulkarni. A Million News Headlines, 2018. URL https://doi.org/10.7910/DVN/ SYBGZL. — [doi:10.7910/dvn/](https://doi.org/10.7910/dvn/) [find](https://scholar.google.com/scholar?q=Rohit+Kulkarni%2E+A+Million+News+Headlines%2C+2018%2E+URL+https%3A%2F%2Fdoi%2Eorg%2F10%2E7910%2FDVN%2F+SYBGZL%2E+%E2%80%94+%5Bdoi%3A10%2E7910%2Fdvn%2F%5D%28https%3A%2F%2Fdoi%2Eorg%2F10%2E7910%2Fdvn%2F%29)
37. Rohit Kulkarni. The Examiner - Spam Clickbait Catalog, 2020a. URL [https://www.kaggle.com/](https://www.kaggle.com/) datasets/therohk/examine-the-examiner. [↗](https://www.kaggle.com/)
38. Rohit Kulkarni. Urban Dictionary Words And Definitions, 2020b. URL [https://www.kaggle.com/datasets/therohk/urban-dictionary-words-dataset](https://www.kaggle.com/datasets/therohk/urban-dictionary-words-dataset). [↗](https://www.kaggle.com/datasets/therohk/urban-dictionary-words-dataset)
39. Rohit Kulkarni. India News Headlines Dataset, 2022. URL [https://www.kaggle.com/datasets/](https://www.kaggle.com/datasets/) therohk/india-headlines-news-dataset. [↗](https://www.kaggle.com/datasets/)
40. Benjamin Letham, Cynthia Rudin, Tyler H. McCormick, and David Madigan. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. The Annals of Applied Statistics, 9(3), Sep 2015. ISSN 1932-6157. doi: 10.1214/15-aoas848. URL http://dx.doi.org/10.1214/15-… [↗](http://dx.doi.org/10.1214/15-…)
41. Derek Lim and Austin R Benson. Expertise and dynamics within crowdsourced musical knowledge curation: A case study of the genius platform. In ICWSM, pp. 373–384, 2021. [find](https://scholar.google.com/scholar?q=Derek+Lim+and+Austin+R+Benson%2E+Expertise+and+dynamics+within+crowdsourced+musical+knowledge+curation%3A+A+case+study+of+the+genius+platform%2E+In+ICWSM%2C+pp%2E+373%E2%80%93384%2C+2021%2E)
42. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. [Wanli: Worker and ai collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/), January 2022. URL [https://arxiv.org/pdf/](https://arxiv.org/pdf/) 2201.05955. [↗](https://arxiv.org/pdf/)
43. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. [Roberta: A robustly optimized bert pretraining approach](https://md.research.elamur.ai/free/en/papers/liu-2019-roberta-robustly-optimized-bert-pretraining/). arXiv preprint arXiv:1907.11692, 2019.
44. Zhi Liu. Reuter_50_50 Data Set, 2011. URL [https://archive.ics.uci.edu/ml/datasets/](https://archive.ics.uci.edu/ml/datasets/) Reuter_50_50. [↗](https://archive.ics.uci.edu/ml/datasets/)
45. Jens Ludwig and Sendhil Mullainathan. Algorithmic behavioral science: Machine learning as a tool for scientific discovery. Chicago Booth Research Paper, (22-15), 2022. [find](https://scholar.google.com/scholar?q=Jens+Ludwig+and+Sendhil+Mullainathan%2E+Algorithmic+behavioral+science%3A+Machine+learning+as+a+tool+for+scientific+discovery%2E+Chicago+Booth+Research+Paper%2C+%2822%2D15%29%2C+2022%2E)
46. Christopher Manning and Hinrich Schutze. Foundations of statistical natural language processing. MIT press, 1999. [find](https://scholar.google.com/scholar?q=Christopher+Manning+and+Hinrich+Schutze%2E+Foundations+of+statistical+natural+language+processing%2E+MIT+press%2C+1999%2E)
47. Ken McGarry. A survey of interestingness measures for knowledge discovery. The knowledge engineering review, 20(1):39–61, 2005. [find](https://scholar.google.com/scholar?q=Ken+McGarry%2E+A+survey+of+interestingness+measures+for+knowledge+discovery%2E+The+knowledge+engineering+review%2C+20%281%29%3A39%E2%80%9361%2C+2005%2E)
48. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. [Pointer sentinel mixture models](https://md.research.elamur.ai/free/en/papers/merity-2016-pointer-sentinel-mixture-models/), 2016.
49. Natan Mish. Federal Reserve Governors Speeches 1996 - 2020, 2020. URL [https://tinyurl.com/3j2e79a6](https://tinyurl.com/3j2e79a6). [↗](https://tinyurl.com/3j2e79a6)
50. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In ACL, 2022. [find](https://scholar.google.com/scholar?q=Swaroop+Mishra%2C+Daniel+Khashabi%2C+Chitta+Baral%2C+and+Hannaneh+Hajishirzi%2E+Cross%2Dtask+generalization+via+natural+language+crowdsourcing+instructions%2E+In+ACL%2C+2022%2E)
51. Rishabh Misra and Prahal Arora. Sarcasm detection using hybrid neural network. arXiv preprint arXiv:1908.07414, 2019. — [arXiv:1908.07414](https://arxiv.org/abs/1908.07414) [find](https://scholar.google.com/scholar?q=Rishabh+Misra+and+Prahal+Arora%2E+Sarcasm+detection+using+hybrid+neural+network%2E+arXiv+preprint+arXiv%3A1908%2E07414%2C+2019%2E+%E2%80%94+%5BarXiv%3A1908%2E07414%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F1908%2E07414%29)
52. Rishabh Misra and Jigyasa Grover. Sculpting Data for ML: The first act of Machine Learning. 01 2021. ISBN 9798585463570. [find](https://scholar.google.com/scholar?q=Rishabh+Misra+and+Jigyasa+Grover%2E+Sculpting+Data+for+ML%3A+The+first+act+of+Machine+Learning%2E+01+2021%2E+ISBN+9798585463570%2E)
53. Nuno Moniz and Luâ C™is Torgo. Multi-source social feedback of online news feeds. CoRR, [Web Link], 2018. [find](https://scholar.google.com/scholar?q=Nuno+Moniz+and+Lu%C3%A2+C%E2%84%A2is+Torgo%2E+Multi%2Dsource+social+feedback+of+online+news+feeds%2E+CoRR%2C+%5BWeb+Link%5D%2C+2018%2E)
54. Mickaël Mouillé. Kickstarter Projects, 2017. URL [https://www.kaggle.com/datasets/](https://www.kaggle.com/datasets/) kemical/kickstarter-projects?select=ks-projects-201612.csv. [↗](https://www.kaggle.com/datasets/)
55. Dong Nguyen, Maria Liakata, Simon DeDeo, Jacob Eisenstein, David Mimno, Rebekah Tromble, and Jane Winters. How we do things with words: Analyzing text as social and cultural data. Frontiers in Artificial Intelligence, 3:62, 2020. [find](https://scholar.google.com/scholar?q=Dong+Nguyen%2C+Maria+Liakata%2C+Simon+DeDeo%2C+Jacob+Eisenstein%2C+David+Mimno%2C+Rebekah+Tromble%2C+and+Jane+Winters%2E+How+we+do+things+with+words%3A+Analyzing+text+as+social+and+cultural+data%2E+Frontiers+in+Artific)
56. Jianmo Ni, Jiacheng Li, and Julian McAuley. [Justifying recommendations using distantly-labeled reviews and fine-grained aspects](https://md.research.elamur.ai/free/en/papers/ni-2019-justifying-recommendations-distantly-labeled-reviews/). In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-…
57. Elle O’Brien. iterative/aita_dataset: Praw rescrape of entire dataset, February 2020. URL https://doi.org/10.5281/zenodo.3677563. — [doi:10.5281/zenodo.3677563](https://doi.org/10.5281/zenodo.3677563) [↗](https://doi.org/10.5281/zenodo.3677563)
58. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. [Training language models to follow instructions with human feedback](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/). arXiv preprint arXiv:2203.02155, 2022.
59. Verónica Pérez-Rosas and Rada Mihalcea. Experiments in open domain deception detection. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1120–1125, 2015. [find](https://scholar.google.com/scholar?q=Ver%C3%B3nica+P%C3%A9rez%2DRosas+and+Rada+Mihalcea%2E+Experiments+in+open+domain+deception+detection%2E+In+Proceedings+of+the+2015+conference+on+empirical+methods+in+natural+language+processing%2C+pp%2E+1120%E2%80%931125%2C+2015%2E)
60. Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, and Mihai Burzo. Deception detection using real-life trial data. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 59–66, 2015. [find](https://scholar.google.com/scholar?q=Ver%C3%B3nica+P%C3%A9rez%2DRosas%2C+Mohamed+Abouelenien%2C+Rada+Mihalcea%2C+and+Mihai+Burzo%2E+Deception+detection+using+real%2Dlife+trial+data%2E+In+Proceedings+of+the+2015+ACM+on+International+Conference+on+Multimodal+Inte)
61. Verónica Pérez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. [Automatic detection of fake news](https://md.research.elamur.ai/free/en/papers/rosas-2018-automatic-detection-fake-news/). arXiv preprint arXiv:1708.07104, 2017.
62. Denis Peskov, Benny Cheng, Ahmed Elgohary, Joe Barrow, Cristian Danescu-Niculescu-Mizil, and Jordan Boyd-Graber. It takes two to lie: One to lie and one to listen. In Association for Computational Linguistics, 2020. [find](https://scholar.google.com/scholar?q=Denis+Peskov%2C+Benny+Cheng%2C+Ahmed+Elgohary%2C+Joe+Barrow%2C+Cristian+Danescu%2DNiculescu%2DMizil%2C+and+Jordan+Boyd%2DGraber%2E+It+takes+two+to+lie%3A+One+to+lie+and+one+to+listen%2E+In+Association+for+Computational+Lin)
63. John P. Pestian, Chris Brew, Pawel Matykiewicz, DJ Hovermale, Neil Johnson, K. Bretonnel Cohen, and Wlodzislaw Duch. A shared task involving multi-label classification of clinical free text. In Biological, translational, and clinical language processing, pp. 97–104, Prague, Czech Republic, June 2007… [find](https://scholar.google.com/scholar?q=John+P%2E+Pestian%2C+Chris+Brew%2C+Pawel+Matykiewicz%2C+DJ+Hovermale%2C+Neil+Johnson%2C+K%2E+Bretonnel+Cohen%2C+and+Wlodzislaw+Duch%2E+A+shared+task+involving+multi%2Dlabel+classification+of+clinical+free+text%2E+In+Biolog)
64. Michael Polanyi, John Ziman, and Steve Fuller. The republic of science: its political and economic theory minerva, i (1)(1962), 54-73. Minerva, 38(1):1–32, 2000. [find](https://scholar.google.com/scholar?q=Michael+Polanyi%2C+John+Ziman%2C+and+Steve+Fuller%2E+The+republic+of+science%3A+its+political+and+economic+theory+minerva%2C+i+%281%29%281962%29%2C+54%2D73%2E+Minerva%2C+38%281%29%3A1%E2%80%9332%2C+2000%2E)
65. Ilan Price, Jordan Gifford-Moore, Jory Fleming, Saul Musker, Maayan Roichman, Guillaume Sylvain, Nithum Thain, Lucas Dixon, and Jeffrey Sorensen. Six attributes of unhealthy conversation. arXiv preprint arXiv:2010.07410, 2020. — [arXiv:2010.07410](https://arxiv.org/abs/2010.07410) [find](https://scholar.google.com/scholar?q=Ilan+Price%2C+Jordan+Gifford%2DMoore%2C+Jory+Fleming%2C+Saul+Musker%2C+Maayan+Roichman%2C+Guillaume+Sylvain%2C+Nithum+Thain%2C+Lucas+Dixon%2C+and+Jeffrey+Sorensen%2E+Six+attributes+of+unhealthy+conversation%2E+arXiv+prepri)
66. Demand Progress. Statements of Administration Policy, 2022. URL [https://github.com/unitedstates/statements-of-administration-policy\#](https://github.com/unitedstates/statements-of-administration-policy\#) statements-of-administration-policy. [↗](https://github.com/unitedstates/statements-of-administration-policy\#)
67. PromptCloud. U.S. Technology Jobs on Dice.com, 2017. URL [https://www.kaggle.com/](https://www.kaggle.com/) datasets/PromptCloudHQ/us-technology-jobs-on-dicecom. [↗](https://www.kaggle.com/)
68. WVO Quine. Naturalistic epistemology. Ontological relativity and other essays, pp. 69–90, 1969. [find](https://scholar.google.com/scholar?q=WVO+Quine%2E+Naturalistic+epistemology%2E+Ontological+relativity+and+other+essays%2C+pp%2E+69%E2%80%9390%2C+1969%2E)
69. Quora. Quora Question quora-question-pairs. Pairs, 2017. [find](https://scholar.google.com/scholar?q=Quora%2E+Quora+Question+quora%2Dquestion%2Dpairs%2E+Pairs%2C+2017%2E)
70. URL — [Know What You Don't Know: Unanswerable Questions for SQuAD](https://md.research.elamur.ai/free/en/papers/rajpurkar-2018-know-what-you-don-t/) [https://www.kaggle.com/c/Pranav](https://www.kaggle.com/c/Pranav) Rajpurkar, Robin Jia, and Percy Liang. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822, 2018. [↗](https://www.kaggle.com/c/Pranav)
71. Justin Robischon. Wikipedia Movie Plots, 2019. URL [https://www.kaggle.com/datasets/](https://www.kaggle.com/datasets/) jrobischon/wikipedia-movie-plots. [↗](https://www.kaggle.com/datasets/)
72. Allen Roush and Arvind Balaji. DebateSum: A large-scale argument mining and summarization dataset. In Proceedings of the 7th Workshop on Argument Mining, pp. 1–7, Online, December 2020. Association for Computational Linguistics. URL https://aclanthology.org/2020. argmining-1.1. [↗](https://aclanthology.org/2020)
73. Dudley Shapere. The structure of scientific revolutions. The Philosophical Review, 73(3):383–394, 1964. [find](https://scholar.google.com/scholar?q=Dudley+Shapere%2E+The+structure+of+scientific+revolutions%2E+The+Philosophical+Review%2C+73%283%29%3A383%E2%80%93394%2C+1964%2E)
74. Chandan Singh, John X Morris, Jyoti Aneja, Alexander M Rush, and Jianfeng Gao. Explaining patterns in data with language models via interpretable autoprompting. arXiv preprint arXiv:2210.01848, 2022. — [arXiv:2210.01848](https://arxiv.org/abs/2210.01848) [find](https://scholar.google.com/scholar?q=Chandan+Singh%2C+John+X+Morris%2C+Jyoti+Aneja%2C+Alexander+M+Rush%2C+and+Jianfeng+Gao%2E+Explaining+patterns+in+data+with+language+models+via+interpretable+autoprompting%2E+arXiv+preprint+arXiv%3A2210%2E01848%2C+2022%2E+)
75. Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. [Reasoning with neural tensor networks for knowledge base completion](https://md.research.elamur.ai/free/en/papers/socher-2013-reasoning-neural-tensor-networks-knowledge/). Advances in neural information processing systems, 26, 2013.
76. Alane Suhr, Clara Vania, Nikita Nangia, Maarten Sap, Mark Yatskar, Samuel Bowman, and Yoav Artzi. Crowdsourcing beyond annotation: Case studies in benchmark data collection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pp. 1–6, 2021. [find](https://scholar.google.com/scholar?q=Alane+Suhr%2C+Clara+Vania%2C+Nikita+Nangia%2C+Maarten+Sap%2C+Mark+Yatskar%2C+Samuel+Bowman%2C+and+Yoav+Artzi%2E+Crowdsourcing+beyond+annotation%3A+Case+studies+in+benchmark+data+collection%2E+In+Proceedings+of+the+2021)
77. J Sun. Daily News for Stock Market Prediction, 2017. datasets/aaron7sun/stocknews. [find](https://scholar.google.com/scholar?q=J+Sun%2E+Daily+News+for+Stock+Market+Prediction%2C+2017%2E+datasets%2Faaron7sun%2Fstocknews%2E)
78. URL [https://www.kaggle.com/Jesse](https://www.kaggle.com/Jesse) Thomason, Jivko Sinapov, Maxwell Svetlik, Peter Stone, and Raymond J Mooney. Learning multi-modal grounded linguistic semantics by playing" i spy". In IJCAI, pp. 3477–3483, 2016. [↗](https://www.kaggle.com/Jesse)
79. Andrew Thompson. All the News 1.0, 2019. URL [https://components.one/datasets/](https://components.one/datasets/) all-the-news-articles-dataset. [↗](https://components.one/datasets/)
80. Elsbeth Turcan and Kathleen McKeown. Dreaddit: A reddit dataset for stress analysis in social media. arXiv preprint arXiv:1911.00133, 2019. [find](https://scholar.google.com/scholar?q=Elsbeth+Turcan+and+Kathleen+McKeown%2E+Dreaddit%3A+A+reddit+dataset+for+stress+analysis+in+social+media%2E+arXiv+preprint+arXiv%3A1911%2E00133%2C+2019%2E)
81. Udacity. Armenian Online Job Postings, 2017. URL [https://www.kaggle.com/datasets/](https://www.kaggle.com/datasets/) udacity/armenian-online-job-postings. [↗](https://www.kaggle.com/datasets/)
82. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. [Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/). arXiv preprint arXiv:2306.07899, 2023.
83. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. URL https://arxiv. org/abs/2204.… [find](https://scholar.google.com/scholar?q=Yizhong+Wang%2C+Swaroop+Mishra%2C+Pegah+Alipoormolabashi%2C+Yeganeh+Kordi%2C+Amirreza+Mirzaei%2C+Anjana+Arunkumar%2C+Arjun+Ashok%2C+Arut+Selvan+Dhanasekaran%2C+Atharva+Naik%2C+David+Stap%2C+et+al%2E+Super%2Dnaturalinstructio)
84. Zihan Wang, Jingbo Shang, and Ruiqi Zhong. Goal-driven explainable clustering via language descriptions. arXiv preprint arXiv:2305.13749, 2023. [find](https://scholar.google.com/scholar?q=Zihan+Wang%2C+Jingbo+Shang%2C+and+Ruiqi+Zhong%2E+Goal%2Ddriven+explainable+clustering+via+language+descriptions%2E+arXiv+preprint+arXiv%3A2305%2E13749%2C+2023%2E)
85. Orion Weller and Kevin Seppi. The rJokes dataset: a large scale humor collection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 6136–6141, Marseille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4. URL https://aclanthology.org/202… [↗](https://aclanthology.org/202…)
86. Adina Williams, Nikita Nangia, and Samuel R Bowman. [A broad-coverage challenge corpus for sentence understanding through inference](https://md.research.elamur.ai/free/en/papers/williams-2018-broad-coverage-challenge-corpus-sentence/). arXiv preprint arXiv:1704.05426, 2017.
87. Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, and Furu Wei. [Language models as inductive reasoners](https://md.research.elamur.ai/free/en/papers/yang-2024-language-models-as-inductive-reasoners/). arXiv preprint arXiv:2212.10923, 2022.
88. Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. Guess the instruction! making language models stronger zero-shot learners. arXiv preprint arXiv:2210.02969, 2022. — [arXiv:2210.02969](https://arxiv.org/abs/2210.02969) [find](https://scholar.google.com/scholar?q=Seonghyeon+Ye%2C+Doyoung+Kim%2C+Joel+Jang%2C+Joongbo+Shin%2C+and+Minjoon+Seo%2E+Guess+the+instruction%21+making+language+models+stronger+zero%2Dshot+learners%2E+arXiv+preprint+arXiv%3A2210%2E02969%2C+2022%2E+%E2%80%94+%5BarXiv%3A2210%2E02)
89. Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing differences between text distributions with natural language. In International Conference on Machine Learning, pp. 27099–27116. PMLR, 2022. [find](https://scholar.google.com/scholar?q=Ruiqi+Zhong%2C+Charlie+Snell%2C+Dan+Klein%2C+and+Jacob+Steinhardt%2E+Describing+differences+between+text+distributions+with+natural+language%2E+In+International+Conference+on+Machine+Learning%2C+pp%2E+27099%E2%80%9327116%2E+)
90. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. [Large language models are human-level prompt engineers](https://md.research.elamur.ai/free/en/papers/zhou-2022-large-language-models-human-level/). arXiv preprint arXiv:2211.01910, 2022.
91. Zhiying Zhu, Weixin Liang, and James Zou. Gsclip: A framework for explaining distribution shifts in natural language. arXiv preprint arXiv:2206.15007, 2022. — [arXiv:2206.15007](https://arxiv.org/abs/2206.15007) [find](https://scholar.google.com/scholar?q=Zhiying+Zhu%2C+Weixin+Liang%2C+and+James+Zou%2E+Gsclip%3A+A+framework+for+explaining+distribution+shifts+in+natural+language%2E+arXiv+preprint+arXiv%3A2206%2E15007%2C+2022%2E+%E2%80%94+%5BarXiv%3A2206%2E15007%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F)

*Updated 2 November 2026*