---
title: "Hypothesis-only Biases in Large Language Model-Elicited Natural"
description: "Crowdsourced NLI corpora are known to contain annotation artifacts: cues in the hypothesis alone… 49 references, cited by 2 papers here."
authors: ["Grace Proebsting", "Adam Poliak"]
year: 2024
venue: "arXiv:2410.08996"
source: "https://arxiv.org/abs/2410.08996"
doi: "10.48550/arxiv.2410.08996"
arxiv: "2410.08996"
licence: "arXiv non-exclusive distribution licence (read only, no redistribution)"
licence_url: "http://arxiv.org/licenses/nonexclusive-distrib/1.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/proebsting-2024-hypothesis-only-biases-large-language/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/proebsting-2024-hypothesis-only-biases-large-language/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/) / Hypothesis-only Biases in Large Language Model-Elicited Natural Language Inference

# Hypothesis-only Biases in Large Language Model-Elicited Natural Language Inference

**Authors:** [Grace Proebsting](https://md.research.elamur.ai/free/en/authors/grace-proebsting/) and [Adam Poliak](https://md.research.elamur.ai/free/en/authors/adam-poliak/) · **Year:** 2024 · **Venue:** arXiv:2410.08996 · **Licence:** [arXiv non-exclusive distribution licence (read only, no redistribution)](http://arxiv.org/licenses/nonexclusive-distrib/1.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2410.08996) · [PDF](https://arxiv.org/pdf/2410.08996) · [DOI 10.48550/arxiv.2410.08996](https://doi.org/10.48550/arxiv.2410.08996) · [arXiv 2410.08996](https://arxiv.org/abs/2410.08996) · [OpenAlex](https://openalex.org/W4403443838)

**Citations:** 0 citations (OpenAlex, 2026-10-14) · [2](https://md.research.elamur.ai/free/en/papers/proebsting-2024-hypothesis-only-biases-large-language/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
Crowdsourced NLI corpora are known to contain annotation artifacts: cues in the hypothesis alone reveal the label. This paper asks whether those artifacts disappear when LLMs, rather than crowd workers, write the hypotheses. The authors gave GPT-4, Llama-2 70b Chat, Mistral 7b Instruct and PaLM 2 the original SNLI worker instructions. Each model produced entailed, neutral and contradicting hypotheses for a third of the SNLI training premises and for all dev premises. They hand-checked label quality and dropped PaLM 2 because too many of its labels were wrong. They also confirmed that the outputs were not copied from SNLI. Next, they trained hypothesis-only Naive Bayes and BERT classifiers and mined label-predictive words. The LLM-written data turned out to be even easier to shortcut than SNLI: BERT reached 86–96% accuracy without seeing the premise, and a handful of unigrams was nearly enough. The work warns that LLM-generated training data inherits, and can amplify, the familiar human biases unless it is filtered.

## Key points
- Each LLM dataset has 133,629 training pairs and 6,525 evaluation pairs, generated with temperature 0.75 and top-p 0.9. The models were gpt-4-0613, llama-2-70b-chat, mistral-7b-instruct-v0.1 and chat-bison.
- The authors manually checked 300 pairs per source and agreed with the labels at these rates:
  - SNLI 92.7%
  - GPT-4 94.3%
  - Llama 89.7%
  - Mistral 83.7%
  - PaLM 2 77.0%, which led to its removal
- Mean Jaccard overlap with the matching SNLI hypotheses was only 0.19–0.25, so the models were not reproducing SNLI text.
- In-domain accuracy of the hypothesis-only BERT classifiers was 0.96 for GPT-4, 0.91 for Llama, 0.86 for Mistral and 0.72 for SNLI. Classifiers trained on SNLI scored higher on GPT-4 test data (NB 0.82, BERT 0.88) than on SNLI's own test data (0.64, 0.72).
- The authors listed give-away words where p(label|word) ≥ 0.8. In GPT-4 contradictions, "swimming in a pool" appears more than 10,000 times, and Mistral overuses "sitting on a bench". Several cue words, such as "There" and "Someone", come straight from the prompt.

## Interesting ideas
- A partial-input classifier (here, hypothesis-only) serves as a cheap probe: if it beats the majority baseline by a wide margin, the dataset has shortcuts.
- Plotting accuracy against the number of chi-squared-selected unigram features shows how few surface cues a dataset needs to be "solved".
- Giveaway mining keeps words with p(label|word) ≥ 0.8 and ranks them by frequency, which quickly exposes repeated templates in generated data.
- Some giveaways are words copied from the instruction prompt, so a generator can echo its context into label-correlated patterns.
- For a fair human-versus-LLM comparison, the authors reused the exact crowd instructions and then checked lexical overlap with the human data to rule out memorization.

## For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

- **I1** (learn): Before training on LLM-teacher silver data, fit a model that sees only part of the input, such as the question without the story; high accuracy flags leaky data.
- **I2** (answer): Run this curve on school-course tests and fable QA sets to find items answerable from cue words alone, and keep them out of reasoning evaluations.
- **I3** (learn): Apply the same mining to LLM-teacher outputs per frame or label to catch stock phrases before they bias training.
- **I4** (back): When checking generated or self-play text, compare its vocabulary against the prompt or template and penalize echoed phrases that track the target label.
- **I5** (learn): When an LLM produces silver annotations for treebank sentences, measure overlap with the gold data to detect contamination rather than genuine labeling.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 16,667 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 16,194 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 16,039 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 15,891 works

🧭 Topics (2): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,320 works · [🧮 Reasoning](https://md.research.elamur.ai/free/en/topics/reasoning/) · 1,426 works

## Cite

Cite as: Grace Proebsting and Adam Poliak. 2024. Hypothesis-only Biases in Large Language Model-Elicited Natural Language Inference. arXiv:2410.08996.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{proebsting2024hypothesis,
  title = {{Hypothesis-only Biases in Large Language Model-Elicited Natural Language Inference}},
  author = {Grace Proebsting and Adam Poliak},
  year = {2024},
  doi = {10.48550/arxiv.2410.08996},
  eprint = {2410.08996},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.08996}
}
```

- [← previous Human and System Perspectives on the Expression of Irony: An Analysis of Likelihood Labels and Rationales](https://md.research.elamur.ai/free/en/papers/maladry-2024-human-system-perspectives-expression-irony/)
- [next → Improving Low-Resource Machine Translation via Cross-Linguistic Transfer from Typologically Similar High-Resource Languages](https://md.research.elamur.ai/free/en/papers/boujkian-2024-improving-low-resource-machine-translation/)
- [↑ back to papers of 2024](https://md.research.elamur.ai/free/en/papers/2024/page/12/)

## References

49 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Mérouane Debbah, Étienne Goffinet, Daniel Hesslow, Julien Launay, Quentin Malartic, et al. 2023. [The falcon series of open language models](https://md.research.elamur.ai/free/en/papers/almazrouei-2023-falcon-series-open-language-models/). arXiv preprint arXiv:2311.16867. [↗](https://arxiv.org/abs/2311.16867)
2. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. [Palm 2 technical report](https://md.research.elamur.ai/free/en/papers/anil-2023-palm-2-technical-report/).
3. Hritik Bansal, Karthik Gopalakrishnan, Saket Dingliwal, Sravan Bodapati, Katrin Kirchhoff, and Dan Roth. 2023. Rethinking the role of scale for in-context learning: An interpretability-based case study at 66 billion scale. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11833–11856, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2212.09095)
4. Parikshit Bansal and Amit Sharma. 2023. Large language models as annotators: Enhancing generalization of nlp models at minimal cost. arXiv preprint arXiv:2306.15766. [↗](https://arxiv.org/abs/2306.15766)
5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. [A large annotated corpus for learning natural language inference](https://md.research.elamur.ai/free/en/papers/bowman-2015-large-annotated-corpus-learning-natural/). In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/d15-1075)
6. Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending:strong neural baselines for the ROC story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 616–622, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zheng+Cai%2C+Lifu+Tu%2C+and+Kevin+Gimpel%2E+2017%2E+Pay+attention+to+the+ending%3Astrong+neural+baselines+for+the+ROC+story+cloze+task%2E+In+Proceedings+of+the+55th+Annual+Meeting+of+the+Association+for+Computati)
7. Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. 2023. DISCO: Distilling counterfactuals with large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5514–5528, Toronto, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Zeming+Chen%2C+Qiyue+Gao%2C+Antoine+Bosselut%2C+Ashish+Sabharwal%2C+and+Kyle+Richardson%2E+2023%2E+DISCO%3A+Distilling+counterfactuals+with+large+language+models%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+As)
8. John Chung, Ece Kamar, and Saleema Amershi. 2023. [Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions](https://md.research.elamur.ai/free/en/papers/chung-2023-increasing-diversity-while-maintaining-accuracy/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 575–593, Toronto, Canada. Association for Computational Linguistics.
9. Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world’s first truly open instruction-tuned llm. [find](https://scholar.google.com/scholar?q=Mike+Conover%2C+Matt+Hayes%2C+Ankit+Mathur%2C+Jianwei+Xie%2C+Jun+Wan%2C+Sam+Shah%2C+Ali+Ghodsi%2C+Patrick+Wendell%2C+Matei+Zaharia%2C+and+Reynold+Xin%2E+2023%2E+Free+dolly%3A+Introducing+the+world%E2%80%99s+first+truly+open+instruct)
10. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05, page 177–190, Berlin, Heidelberg. Springer-Verlag. [find](https://scholar.google.com/scholar?q=Ido+Dagan%2C+Oren+Glickman%2C+and+Bernardo+Magnini%2E+2005%2E+The+pascal+recognising+textual+entailment+challenge%2E+In+Proceedings+of+the+First+International+Conference+on+Machine+Learning+Challenges%3A+Evaluati)
11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. [BERT: Pre-training of deep bidirectional transformers for language understanding](https://md.research.elamur.ai/free/en/papers/devlin-2019-bert-pre-training-deep-bidirectional/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n19-1423)
12. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2021. A mathematical framework for transformer circuits. Transformer Circuits Thread. Https://transformer-circuits.pub/2021/framework/index.html. [find](https://scholar.google.com/scholar?q=Nelson+Elhage%2C+Neel+Nanda%2C+Catherine+Olsson%2C+Tom+Henighan%2C+Nicholas+Joseph%2C+Ben+Mann%2C+Amanda+Askell%2C+Yuntao+Bai%2C+Anna+Chen%2C+Tom+Conerly%2C+Nova+DasSarma%2C+Dawn+Drain%2C+Deep+Ganguli%2C+Zac+Hatfield%2DDodds%2C+Da)
13. Shi Feng, Eric Wallace, and Jordan Boyd-Graber. 2019. Misleading failures of partial-input baselines. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5533– 5538, Florence, Italy. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shi+Feng%2C+Eric+Wallace%2C+and+Jordan+Boyd%2DGraber%2E+2019%2E+Misleading+failures+of+partial%2Dinput+baselines%2E+In+Proceedings+of+the+57th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+pages+)
14. Johann Frei and Frank Kramer. 2023. Annotated dataset creation through large language models for non-english medical nlp. Journal of Biomedical Informatics, page 104478. [↗](https://doi.org/10.1016/j.jbi.2023.104478)
15. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. [Chatgpt outperforms crowd-workers for text-annotation tasks](https://md.research.elamur.ai/free/en/papers/gilardi-2023-chatgpt-outperforms-crowd-workers-text/). arXiv preprint arXiv:2303.15056. [↗](https://arxiv.org/abs/2303.15056)
16. Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. 2024. Olmo: Accelerating the science of language models. [↗](https://doi.org/10.48550/arxiv.2402.00838)
17. Himanshu Gupta, Kevin Scaria, Ujjwala Anantheswaran, Shreyas Verma, Mihir Parmar, Saurabh Arjun Sawant, Swaroop Mishra, and Chitta Baral. 2023. Targen: Targeted data generation with large language models. arXiv preprint arXiv:2310.17876. [↗](https://arxiv.org/abs/2310.17876)
18. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. [Annotation artifacts in natural language inference data](https://md.research.elamur.ai/free/en/papers/gururangan-2018-annotation-artifacts-natural-language-inference/). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/n18-2017)
19. Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, et al. 2023. [Annollm: Making large language models to be better crowdsourced annotators](https://md.research.elamur.ai/free/en/papers/he-2023-annollm-making-large-language-models/). arXiv preprint arXiv:2303.16854. [↗](https://arxiv.org/abs/2303.16854)
20. Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2023. [Unnatural instructions: Tuning language models with (almost) no human labor](https://md.research.elamur.ai/free/en/papers/honovich-2023-unnatural-instructions-tuning-language-models/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14409–14428, Toronto, Canada. Association for Computational Linguistics.
21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7b. [find](https://scholar.google.com/scholar?q=Albert+Q%2E+Jiang%2C+Alexandre+Sablayrolles%2C+Arthur+Mensch%2C+Chris+Bamford%2C+Devendra+Singh+Chaplot%2C+Diego+de+las+Casas%2C+Florian+Bressand%2C+Gianna+Lengyel%2C+Guillaume+Lample%2C+Lucile+Saulnier%2C+L%C3%A9lio+Renard+Lav)
22. Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. 2023. Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution. [↗](https://doi.org/10.48550/arxiv.2307.16883)
23. Divyansh Kaushik and Zachary C. Lipton. 2018. [How much reading does reading comprehension require? a critical investigation of popular benchmarks](https://md.research.elamur.ai/free/en/papers/kaushik-2018-how-much-reading-does-reading/). In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010– 5015, Brussels, Belgium. Association for Computational Linguistics.
24. Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022. [WANLI: Worker and AI collaboration for natural language inference dataset creation](https://md.research.elamur.ai/free/en/papers/liu-2022-wanli-worker-ai-collaboration-natural/). In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
25. Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations. [↗](https://doi.org/10.48550/arxiv.1711.05101)
26. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. [In-context learning and induction heads](https://md.research.elamur.ai/free/en/papers/olsson-2022-context-learning-induction-heads/). Transformer Circuits Thread. Https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/index.html. [↗](https://doi.org/10.48550/arxiv.2209.11895)
27. OpenAI. 2023. [Gpt-4 technical report](https://md.research.elamur.ai/free/en/papers/openai-2023-gpt-4-technical-report/).
28. F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830. [↗](https://doi.org/10.48550/arxiv.1201.0490)
29. Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2023. [Instruction tuning with gpt-4](https://md.research.elamur.ai/free/en/papers/peng-2023-instruction-tuning-gpt-4/). [↗](https://doi.org/10.48550/arxiv.2304.03277)
30. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. [Hypothesis only baselines in natural language inference](https://md.research.elamur.ai/free/en/papers/poliak-2018-hypothesis-only-baselines-natural-language/). In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/s18-2023)
31. Gaurav Sahu, Pau Rodriguez, Issam Laradji, Parmida Atighehchian, David Vazquez, and Dzmitry Bahdanau. 2022. [Data augmentation for intent classification with off-the-shelf large language models](https://md.research.elamur.ai/free/en/papers/sahu-2022-data-augmentation-intent-classification-off/). In Proceedings of the 4th Workshop on NLP for Conversational AI, pages 47–57, Dublin, Ireland. Association for Computational Linguistics.
32. Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith. 2017. The effect of different writing tasks on linguistic style: A case study of the ROC story cloze task. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 15–25, Vancouver, Canada. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Roy+Schwartz%2C+Maarten+Sap%2C+Ioannis+Konstas%2C+Leila+Zilles%2C+Yejin+Choi%2C+and+Noah+A%2E+Smith%2E+2017%2E+The+effect+of+different+writing+tasks+on+linguistic+style%3A+A+case+study+of+the+ROC+story+cloze+task%2E+In+P)
33. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. [Llama 2: Open foundation and fine-tuned chat models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-2-open-foundation-fine/).
34. Masatoshi Tsuchiya. 2018. [Performance impact caused by hidden bias of training data for recognizing textual entailment](https://md.research.elamur.ai/free/en/papers/tsuchiya-2018-performance-impact-caused-hidden-bias/). In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
35. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. [Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks](https://md.research.elamur.ai/free/en/papers/veselovsky-2023-artificial-artificial-artificial-intelligence-crowd/). arXiv preprint arXiv:2306.07899. [↗](https://arxiv.org/abs/2306.07899)
36. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. [Superglue: A stickier benchmark for general-purpose language understanding systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/). Advances in neural information processing systems, 32.
37. Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model. [find](https://scholar.google.com/scholar?q=Ben+Wang+and+Aran+Komatsuzaki%2E+2021%2E+Gpt%2Dj%2D6b%3A+A+6+billion+parameter+autoregressive+language+model%2E)
38. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2212.10560)
39. Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022. Reframing human-AI collaboration for generating free-text explanations. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 632–658, Seattle, United States. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2112.08674)
40. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. [Transformers: State-of-the-art natural language processing](https://md.research.elamur.ai/free/en/papers/wolf-2020-transformers-state-art-natural-language/). In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. [↗](https://doi.org/10.18653/v1/2020.emnlp-demos.6)
41. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, et al. 2022. Bloom: A 176bparameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [↗](https://arxiv.org/abs/2211.05100)
42. Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021. [Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models](https://md.research.elamur.ai/free/en/papers/wu-2021-polyjuice-generating-counterfactuals-explaining-evaluating/). In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6707–6723, Online. Association for Computational Linguistics. [↗](https://doi.org/10.48550/arxiv.2101.00288)
43. Canwen Xu, Corby Rosset, Luciano Del Corro, Shweti Mahajan, Julian McAuley, Jennifer Neville, Ahmed Hassan Awadallah, and Nikhil Rao. 2023a. Contrastive post-training large language models on data curriculum. [find](https://scholar.google.com/scholar?q=Canwen+Xu%2C+Corby+Rosset%2C+Luciano+Del+Corro%2C+Shweti+Mahajan%2C+Julian+McAuley%2C+Jennifer+Neville%2C+Ahmed+Hassan+Awadallah%2C+and+Nikhil+Rao%2E+2023a%2E+Contrastive+post%2Dtraining+large+language+models+on+data+cur)
44. Ran Xu, Hejie Cui, Yue Yu, Xuan Kan, Wenqi Shi, Yuchen Zhuang, Wei Jin, Joyce Ho, and Carl Yang. 2023b. Knowledge-infused prompting: Assessing and advancing clinical text data generation with large language models. [find](https://scholar.google.com/scholar?q=Ran+Xu%2C+Hejie+Cui%2C+Yue+Yu%2C+Xuan+Kan%2C+Wenqi+Shi%2C+Yuchen+Zhuang%2C+Wei+Jin%2C+Joyce+Ho%2C+and+Carl+Yang%2E+2023b%2E+Knowledge%2Dinfused+prompting%3A+Assessing+and+advancing+clinical+text+data+generation+with+large+la)
45. Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Lingpeng Kong. 2022. [ZeroGen: Efficient zero-shot learning via dataset generation](https://md.research.elamur.ai/free/en/papers/ye-2022-zerogen-efficient-zero-shot-learning/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11653–11669, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
46. Asaf Yehudai, Boaz Carmeli, Yosi Mass, Ofir Arviv, Nathaniel Mills, Assaf Toledo, Eyal Shnarch, and Leshem Choshen. 2024. Genie: Achieving human parity in content-grounded datasets generation. arXiv preprint arXiv:2401.14367. [↗](https://arxiv.org/abs/2401.14367)
47. Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B. Hashimoto. 2024. [Benchmarking Large Language Models for News Summarization](https://md.research.elamur.ai/free/en/papers/zhang-2024-benchmarking-large-language-models-news/). Transactions of the Association for Computational Linguistics, 12:39–57. [↗](https://doi.org/10.1162/tacl_a_00632)
48. Mengjie Zhao, Fei Mi, Yasheng Wang, Minglei Li, Xin Jiang, Qun Liu, and Hinrich Schuetze. 2022. LM-Turk: Few-shot learners as crowdsourcing workers in a language-model-as-a-service framework. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 675–692, Seattle, United States. Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Mengjie+Zhao%2C+Fei+Mi%2C+Yasheng+Wang%2C+Minglei+Li%2C+Xin+Jiang%2C+Qun+Liu%2C+and+Hinrich+Schuetze%2E+2022%2E+LM%2DTurk%3A+Few%2Dshot+learners+as+crowdsourcing+workers+in+a+language%2Dmodel%2Das%2Da%2Dservice+framework%2E+In+Findi)
49. Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024. [Can Large Language Models Transform Computational Social Science?](https://md.research.elamur.ai/free/en/papers/ziems-2023-large-language-models-transform-computational/) Computational Linguistics, pages 1–55. [↗](https://doi.org/10.1162/coli_a_00502)

*Updated 1 November 2026*