---
title: "Task Contamination — paper by Li and Flanigan (2023)"
description: "The authors call this task contamination and distinguish it from leakage of test sets. 48 references, cited by 46 papers here."
authors: ["Changmao Li", "Jeffrey Flanigan"]
year: 2023
venue: "arXiv:2312.16337"
source: "https://arxiv.org/abs/2312.16337"
doi: "10.48550/arxiv.2312.16337"
arxiv: "2312.16337"
licence: "CC BY 4.0 (arXiv)"
licence_url: "http://creativecommons.org/licenses/by/4.0/"
summary_licence: "Elamur Cosmic License"
canonical: "https://research.elamur.ai/free/en/papers/li-2023-task-contamination-language-models-may/"
---
[Elamur Research](https://md.research.elamur.ai/free/en/) · [this page in HTML](https://research.elamur.ai/free/en/papers/li-2023-task-contamination-language-models-may/)

[home](https://md.research.elamur.ai/free/en/) / [papers](https://md.research.elamur.ai/free/en/papers/) / [2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/) / Task Contamination: Language Models May Not Be Few-Shot Anymore

# Task Contamination: Language Models May Not Be Few-Shot Anymore

**Authors:** [Changmao Li](https://md.research.elamur.ai/free/en/authors/changmao-li/) and [Jeffrey Flanigan](https://md.research.elamur.ai/free/en/authors/jeffrey-flanigan/) · **Year:** 2023 · **Venue:** [arXiv:2312.16337](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/) ([Proceedings of the AAAI Conference on Artificial Intelligence 2023](https://md.research.elamur.ai/free/en/venues/proceedings-of-the-aaai-conference-on-artificial-intelligence/2023/)) · **Publisher:** [Association for the Advancement of Artificial Intelligence](https://md.research.elamur.ai/free/en/publishers/association-for-the-advancement-of-artificial-intelligence/) · **Licence:** [CC BY 4.0 (arXiv)](http://creativecommons.org/licenses/by/4.0/)

**Links:** [Publisher's page](https://arxiv.org/abs/2312.16337) · [PDF](https://arxiv.org/pdf/2312.16337) · [DOI 10.48550/arxiv.2312.16337](https://doi.org/10.48550/arxiv.2312.16337) · [arXiv 2312.16337](https://arxiv.org/abs/2312.16337) · [OpenAlex](https://openalex.org/W4393160461)

**Citations:** 43 citations (OpenAlex, 2026-10-14) · [46](https://md.research.elamur.ai/free/en/papers/li-2023-task-contamination-language-models-may/cited-by/) in our library, of the 149,439 papers whose bibliographies we read

## Summary
LLMs are often credited with solving tasks from zero or only a handful of examples. That claim fails if the pre-training data already contained labelled training examples of the evaluated task. The authors call this task contamination and distinguish it from leakage of test sets. Li and Flanigan study the problem systematically. They test 12 models (five closed GPT-3 series versions and seven open models) on 16 classification datasets plus the Spider semantic parsing task. They combine four complementary detection strategies:
- a chronological comparison of datasets published before versus after each model's data-collection cutoff;
- searching the pre-training corpora;
- prompting instruction-tuned models to generate task examples;
- a membership inference check based on exact output matches.

Models beat the majority baseline far more often on older datasets than on newer ones. On tasks where contamination is impossible, they seldom beat that baseline significantly, either zero-shot or few-shot. The study suggests many reported few-shot gains may reflect prior exposure rather than genuine generalization. Benchmark results should therefore be interpreted with dataset release dates in mind.

## Key points
- **Models:** davinci, text-davinci-001/002/003 and GPT-3.5-turbo-0301 (closed), plus Fairseq MoE, GPT-J, OPT, BLOOM, LLaMA, Alpaca and Vicuna (open). Training-data cutoffs range from early 2019 to late 2021 or later.
- **Datasets by release year:** BoolQ, WiC, CB, QNLI, MRPC, SST-2, COPA, WNLI and RTE make up the older group (2009–2019). The newer group (2021–2023) is StrategyQA, NewsMTSC-MT, NewsMTSC-RW, NLI4Wills, CREPE, FOMC and NewsMet.
- **Chronological analysis:** of 192 model–dataset pairs, 136 fall before the model's data cutoff and 56 after. The gap in how often models beat the majority baseline is significant at the 99% level (Mann-Whitney U test). Per-pair significance was checked with a t-test.
- **Spider:** the number of extracted task examples correlates with model accuracy at R = 0.88. Across GPT-3 versions, extractable examples grew from davinci to GPT-3.5-turbo, matching the rise in zero-shot scores.
- **Method trade-off:** corpus inspection, example extraction and membership inference are precise but miss many cases. Chronological analysis catches more cases but is less precise. This is why the authors use all four together.

*Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.*

## Institutions

- [University of California, Santa Cruz](https://md.research.elamur.ai/free/en/universities/university-of-california-santa-cruz/) — University, Santa Cruz, United States · 38 works · 3,604 citations

📚 Shelves (4): [🖥️ Computer Science](https://md.research.elamur.ai/free/en/shelves/computer-science/) · 18,614 works · [🤖 Artificial Intelligence](https://md.research.elamur.ai/free/en/shelves/computer-science/artificial-intelligence/) · 17,965 works · [🎭 Arts and Humanities](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/) · 17,450 works · [🗣️ Language and Linguistics](https://md.research.elamur.ai/free/en/shelves/arts-and-humanities/language-and-linguistics/) · 17,296 works

🧭 Topics (1): [🧠 Language models & pretraining](https://md.research.elamur.ai/free/en/topics/language-models/) · 3,894 works

## Cite

Cite as: Changmao Li and Jeffrey Flanigan. 2023. Task Contamination: Language Models May Not Be Few-Shot Anymore. arXiv:2312.16337.

The paper itself, not this page. BibTeX from the library's record of the work.

```bibtex
@misc{li2023task,
  title = {{Task Contamination: Language Models May Not Be Few-Shot Anymore}},
  author = {Changmao Li and Jeffrey Flanigan},
  year = {2023},
  doi = {10.48550/arxiv.2312.16337},
  eprint = {2312.16337},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.16337}
}
```

- [← previous Safety Assessment of Chinese Large Language Models](https://md.research.elamur.ai/free/en/papers/sun-2023-safety-assessment-chinese-large-language/)
- [next → Landmark Attention: Random-Access Infinite Context Length for Transformers](https://md.research.elamur.ai/free/en/papers/mohtashami-2023-landmark-attention-random-access-infinite/)
- [↑ back to papers of 2023](https://md.research.elamur.ai/free/en/papers/2023/page/8/)

## References

48 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aiyappa, R.; An, J.; Kwak, H.; and Ahn, Y.-Y. 2023. [Can we trust the evaluation on ChatGPT?](https://md.research.elamur.ai/free/en/papers/aiyappa-2023-can-we-trust-evaluation-chatgpt/) arXiv:2303.12767.
2. Artetxe, M.; Bhosale, S.; Goyal, N.; Mihaylov, T.; Ott, M.; Shleifer, S.; Lin, X. V.; Du, J.; Iyer, S.; Pasunuru, R.; Anantharaman, G.; Li, X.; Chen, S.; Akin, H.; Baines, M.; Martin, L.; Zhou, X.; Koura, P. S.; O’Horo, B.; Wang, J.; Zettlemoyer, L.; Diab, M.; Kozareva, Z.; and Stoyanov, V. 2022. Ef… — [Efficient Large Scale Language Modeling with Mixtures of Experts](https://md.research.elamur.ai/free/en/papers/artetxe-2021-efficient-large-scale-language-modeling/)
3. Li, Y. 2023. Estimating Contamination via Perplexity: Quantifying Memorisation in Language Model Evaluation. arXiv:2309.10677. — [arXiv:2309.10677](https://arxiv.org/abs/2309.10677) [find](https://scholar.google.com/scholar?q=Li%2C+Y%2E+2023%2E+Estimating+Contamination+via+Perplexity%3A+Quantifying+Memorisation+in+Language+Model+Evaluation%2E+arXiv%3A2309%2E10677%2E+%E2%80%94+%5BarXiv%3A2309%2E10677%5D%28https%3A%2F%2Farxiv%2Eorg%2Fabs%2F2309%2E10677%29)
4. Magar, I.; and Schwartz, R. 2022. Data Contamination: From Memorization to Exploitation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 157–165. Dublin, Ireland: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Magar%2C+I%2E%3B+and+Schwartz%2C+R%2E+2022%2E+Data+Contamination%3A+From+Memorization+to+Exploitation%2E+In+Proceedings+of+the+60th+Annual+Meeting+of+the+Association+for+Computational+Linguistics%2C+157%E2%80%93165%2E+Dublin%2C+Ir)
5. Mann, H. B.; and Whitney, D. R. 1947. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1): 50– 60. [find](https://scholar.google.com/scholar?q=Mann%2C+H%2E+B%2E%3B+and+Whitney%2C+D%2E+R%2E+1947%2E+On+a+Test+of+Whether+one+of+Two+Random+Variables+is+Stochastically+Larger+than+the+Other%2E+The+Annals+of+Mathematical+Statistics%2C+18%281%29%3A+50%E2%80%93+60%2E)
6. OpenAI. 2023a. OpenAI Examples. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023a%2E+OpenAI+Examples%2E)
7. OpenAI. 2023b. OpenAI Models. [find](https://scholar.google.com/scholar?q=OpenAI%2E+2023b%2E+OpenAI+Models%2E)
8. Oren, Y.; Meister, N.; Chatterji, N.; Ladhak, F.; and Hashimoto, T. B. 2023. [Proving Test Set Contamination in Black Box Language Models](https://md.research.elamur.ai/free/en/papers/oren-2023-proving-test-set-contamination-black/). arXiv:2310.17623.
9. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Gray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. [Training language models to follow instructions…](https://md.research.elamur.ai/free/en/papers/ouyang-2022-training-language-models-follow-instructions/)
10. Pilehvar, M. T.; and Camacho-Collados, J. 2019. [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://md.research.elamur.ai/free/en/papers/pilehvar-2019-wic-word-context-dataset-evaluating/). In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1267–1273. Min…
11. Poesia, G.; Polozov, A.; Le, V.; Tiwari, A.; Soares, G.; Meek, C.; and Gulwani, S. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. In International Conference on Learning Representations. [find](https://scholar.google.com/scholar?q=Poesia%2C+G%2E%3B+Polozov%2C+A%2E%3B+Le%2C+V%2E%3B+Tiwari%2C+A%2E%3B+Soares%2C+G%2E%3B+Meek%2C+C%2E%3B+and+Gulwani%2C+S%2E+2022%2E+Synchromesh%3A+Reliable+Code+Generation+from+Pre%2Dtrained+Language+Models%2E+In+International+Conference+on+Learning)
12. Qin, C.; Zhang, A.; Zhang, Z.; Chen, J.; Yasunaga, M.; and Yang, D. 2023. [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://md.research.elamur.ai/free/en/papers/qin-2023-chatgpt-general-purpose-natural-language/) Qin, G.; and Eisner, J. 2021. Learning How to Ask: Querying LMs with Mixtures of Soft Prompts. In Proceedings of the 2021 Conference of the North American C…
13. Roemmele, M.; Bejan, C. A.; and Gordon, A. S. 2011. Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, 90–95. [find](https://scholar.google.com/scholar?q=Roemmele%2C+M%2E%3B+Bejan%2C+C%2E+A%2E%3B+and+Gordon%2C+A%2E+S%2E+2011%2E+Choice+of+Plausible+Alternatives%3A+An+Evaluation+of+Commonsense+Causal+Reasoning%2E+In+AAAI+spring+symposium%3A+logical+formalizations+of+commonsense+rea)
14. Sainz, O.; Campos, J.; García-Ferrero, I.; Etxaniz, J.; de Lacalle, O. L.; and Agirre, E. 2023a. [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://md.research.elamur.ai/free/en/papers/sainz-2023-nlp-evaluation-trouble-need-measure/). In Bouamor, H.; Pino, J.; and Bali, K., eds., Findings of the Association for Computational Linguistics: EMNLP…
15. Dolan, W. B.; and Brockett, C. 2005. [Automatically Constructing a Corpus of Sentential Paraphrases](https://md.research.elamur.ai/free/en/papers/dolan-2005-automatically-constructing-corpus-sentential-paraphrases/). In Proceedings of the 3rd International Workshop on Paraphrasing (IWP2005).
16. Geva, M.; Khashabi, D.; Segal, E.; Khot, T.; Roth, D.; and Berant, J. 2021. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. Transactions of the Association for Computational Linguistics, 9: 346–361. [find](https://scholar.google.com/scholar?q=Geva%2C+M%2E%3B+Khashabi%2C+D%2E%3B+Segal%2C+E%2E%3B+Khot%2C+T%2E%3B+Roth%2C+D%2E%3B+and+Berant%2C+J%2E+2021%2E+Did+Aristotle+Use+a+Laptop%3F+A+Question+Answering+Benchmark+with+Implicit+Reasoning+Strategies%2E+Transactions+of+the+Associati)
17. Giampiccolo, D.; Dang, H. T.; Magnini, B.; Dagan, I.; Cabrio, E.; and Dolan, W. B. 2008. The Fourth PASCAL Recognizing Textual Entailment Challenge. In Text Analysis Conference. [find](https://scholar.google.com/scholar?q=Giampiccolo%2C+D%2E%3B+Dang%2C+H%2E+T%2E%3B+Magnini%2C+B%2E%3B+Dagan%2C+I%2E%3B+Cabrio%2C+E%2E%3B+and+Dolan%2C+W%2E+B%2E+2008%2E+The+Fourth+PASCAL+Recognizing+Textual+Entailment+Challenge%2E+In+Text+Analysis+Conference%2E)
18. Golchin, S.; and Surdeanu, M. 2023. [Time Travel in LLMs: Tracing Data Contamination in Large Language Models](https://md.research.elamur.ai/free/en/papers/golchin-2023-time-travel-llms-tracing-data/). arXiv:2308.08493.
19. Hamborg, F.; and Donnay, K. 2021. NewsMTSC: A Dataset for (Multi-)Target-dependent Sentiment Classification in Political News Articles. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 1663–1675. Online: Association for Comp… [find](https://scholar.google.com/scholar?q=Hamborg%2C+F%2E%3B+and+Donnay%2C+K%2E+2021%2E+NewsMTSC%3A+A+Dataset+for+%28Multi%2D%29Target%2Ddependent+Sentiment+Classification+in+Political+News+Articles%2E+In+Proceedings+of+the+16th+Conference+of+the+European+Chapter+of)
20. Hu, H.; Salcic, Z.; Sun, L.; Dobbie, G.; Yu, P. S.; and Zhang, X. 2022a. Membership inference attacks on machine learning: A survey. ACM Computing Surveys (CSUR), 54(11s): 1–37. [find](https://scholar.google.com/scholar?q=Hu%2C+H%2E%3B+Salcic%2C+Z%2E%3B+Sun%2C+L%2E%3B+Dobbie%2C+G%2E%3B+Yu%2C+P%2E+S%2E%3B+and+Zhang%2C+X%2E+2022a%2E+Membership+inference+attacks+on+machine+learning%3A+A+survey%2E+ACM+Computing+Surveys+%28CSUR%29%2C+54%2811s%29%3A+1%E2%80%9337%2E)
21. Hu, Y.; Lee, C.-H.; Xie, T.; Yu, T.; Smith, N. A.; and Ostendorf, M. 2022b. [In-Context Learning for Few-Shot Dialogue State Tracking](https://md.research.elamur.ai/free/en/papers/hu-2022-context-learning-few-shot-dialogue/). In Findings of the Association for Computational Linguistics: EMNLP 2022, 2627–2643. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
22. Jacovi, A.; Caciularu, A.; Goldman, O.; and Goldberg, Y. 2023. Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks. arXiv:2305.10160. [find](https://scholar.google.com/scholar?q=Jacovi%2C+A%2E%3B+Caciularu%2C+A%2E%3B+Goldman%2C+O%2E%3B+and+Goldberg%2C+Y%2E+2023%2E+Stop+Uploading+Test+Data+in+Plain+Text%3A+Practical+Strategies+for+Mitigating+Data+Contamination+by+Evaluation+Benchmarks%2E+arXiv%3A2305%2E10160)
23. Joseph, R.; Liu, T.; Ng, A. B.; See, S.; and Rai, S. 2023. NewsMet : A ‘do it all’ Dataset of Contemporary Metaphors in News Headlines. In Findings of the Association for Computational Linguistics: ACL 2023, 10090–10104. Toronto, Canada: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Joseph%2C+R%2E%3B+Liu%2C+T%2E%3B+Ng%2C+A%2E+B%2E%3B+See%2C+S%2E%3B+and+Rai%2C+S%2E+2023%2E+NewsMet+%3A+A+%E2%80%98do+it+all%E2%80%99+Dataset+of+Contemporary+Metaphors+in+News+Headlines%2E+In+Findings+of+the+Association+for+Computational+Linguistics%3A+AC)
24. Kwak, A.; Israelsen, J.; Morrison, C.; Bambauer, D.; and Surdeanu, M. 2022. Validity Assessment of Legal Will Statements as Natural Language Inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, 6047–6056. Abu Dhabi, United Arab Emirates: Association for Computational… [find](https://scholar.google.com/scholar?q=Kwak%2C+A%2E%3B+Israelsen%2C+J%2E%3B+Morrison%2C+C%2E%3B+Bambauer%2C+D%2E%3B+and+Surdeanu%2C+M%2E+2022%2E+Validity+Assessment+of+Legal+Will+Statements+as+Natural+Language+Inference%2E+In+Findings+of+the+Association+for+Computational)
25. Lester, B.; Al-Rfou, R.; and Constant, N. 2021. [The Power of Scale for Parameter-Efficient Prompt Tuning](https://md.research.elamur.ai/free/en/papers/lester-2021-power-scale-parameter-efficient-prompt/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3045–3059. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
26. Levesque, H. J.; Davis, E.; and Morgenstern, L. 2012. The Winograd schema challenge. [find](https://scholar.google.com/scholar?q=Levesque%2C+H%2E+J%2E%3B+Davis%2C+E%2E%3B+and+Morgenstern%2C+L%2E+2012%2E+The+Winograd+schema+challenge%2E)
27. D.; Datta, D.; Ganguli, D.; Emelin, D.; Kleyko, D.; Yuret, D.; Chen, D.; Tam, D.; Hupkes, D.; Misra, D.; Buzan, D.; Mollo, D. C.; Yang, D.; Lee, D.-H.; Schrader, D.; Shutova, E.; Cubuk, E. D.; Segal, E.; Hagerman, E.; Barnes, E.; Donoway, E.; Pavlick, E.; Rodolà, E.; Lam, E.; Chu, E.; Tang, E.; Erde… [find](https://scholar.google.com/scholar?q=D%2E%3B+Datta%2C+D%2E%3B+Ganguli%2C+D%2E%3B+Emelin%2C+D%2E%3B+Kleyko%2C+D%2E%3B+Yuret%2C+D%2E%3B+Chen%2C+D%2E%3B+Tam%2C+D%2E%3B+Hupkes%2C+D%2E%3B+Misra%2C+D%2E%3B+Buzan%2C+D%2E%3B+Mollo%2C+D%2E+C%2E%3B+Yang%2C+D%2E%3B+Lee%2C+D%2E%2DH%2E%3B+Schrader%2C+D%2E%3B+Shutova%2C+E%2E%3B+Cubuk%2C+E%2E+D%2E%3B+Segal%2C+)
28. Scao, T. L.; Fan, A.; Akiki, C.; Pavlick, E.; Ilić, S.; Hesslow, D.; Castagné, R.; Luccioni, A. S.; Yvon, F.; Gallé, M.; et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. [find](https://scholar.google.com/scholar?q=Scao%2C+T%2E+L%2E%3B+Fan%2C+A%2E%3B+Akiki%2C+C%2E%3B+Pavlick%2C+E%2E%3B+Ili%C4%87%2C+S%2E%3B+Hesslow%2C+D%2E%3B+Castagn%C3%A9%2C+R%2E%3B+Luccioni%2C+A%2E+S%2E%3B+Yvon%2C+F%2E%3B+Gall%C3%A9%2C+M%2E%3B+et+al%2E+2022%2E+Bloom%3A+A+176b%2Dparameter+open%2Daccess+multilingual+language+model%2E+a)
29. Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. [Toolformer: Language Models Can Teach Themselves to Use Tools](https://md.research.elamur.ai/free/en/papers/schick-2023-toolformer-language-models-can-teach/).
30. Schick, T.; and Schütze, H. 2021a. [Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference](https://md.research.elamur.ai/free/en/papers/schick-2021-exploiting-cloze-questions-few-shot/). In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 255–269. Online: Association for Computational L…
31. Schick, T.; and Schütze, H. 2021b. [Few-Shot Text Generation with Natural Language Instructions](https://md.research.elamur.ai/free/en/papers/schick-2021-few-shot-text-generation-natural/). In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 390–402. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
32. Shah, A.; Paturi, S.; and Chava, S. 2023. Trillion Dollar Words: A New Financial Dataset, Task & Market Analysis. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 6664–6679. Toronto, Canada: Association for Computational Linguistics. [find](https://scholar.google.com/scholar?q=Shah%2C+A%2E%3B+Paturi%2C+S%2E%3B+and+Chava%2C+S%2E+2023%2E+Trillion+Dollar+Words%3A+A+New+Financial+Dataset%2C+Task+%26+Market+Analysis%2E+In+Proceedings+of+the+61st+Annual+Meeting+of+the+Association+for+Computational+Linguis)
33. Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A.; and Potts, C. 2013. [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank](https://md.research.elamur.ai/free/en/papers/wei-2022-chain-thought-prompting-elicits-reasoning/). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1631–1642. Seattle, Washington, USA: As…
34. Yang, R.; Song, L.; Li, Y.; Zhao, S.; Ge, Y.; Li, X.; and Shan, Y. 2023. [GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction](https://md.research.elamur.ai/free/en/papers/yang-2023-gpt4tools-teaching-large-language-model/). arXiv:2305.18752.
35. Yu, T.; Zhang, R.; Yang, K.; Yasunaga, M.; Wang, D.; Li, Z.; Ma, J.; Li, I.; Yao, Q.; Roman, S.; Zhang, Z.; and Radev, D. 2018. [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://md.research.elamur.ai/free/en/papers/yu-2018-spider-large-scale-human-labeled/). In Proceedings of the 2018 Conference on Empirical Methods i…
36. Yu, X.; Min, S.; Zettlemoyer, L.; and Hajishirzi, H. 2023. [CREPE: Open-Domain Question Answering with False Presuppositions](https://md.research.elamur.ai/free/en/papers/yu-2023-crepe-open-domain-question-answering/). In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 10457– 10480. Toronto, Canada: Association for Computational Linguistics.
37. Zhang, S.; Roller, S.; Goyal, N.; Artetxe, M.; Chen, M.; Chen, S.; Dewan, C.; Diab, M.; Li, X.; Lin, X. V.; Mihaylov, T.; Ott, M.; Shleifer, S.; Shuster, K.; Simig, D.; Koura, P. S.; Sridhar, A.; Wang, T.; and Zettlemoyer, L. 2022. [OPT: Open Pre-trained Transformer Language Models](https://md.research.elamur.ai/free/en/papers/zhang-2022-opt-open-pre-trained-transformer/). arXiv:2205.01068.
38. Zhou, K.; Zhu, Y.; Chen, Z.; Chen, W.; Zhao, W. X.; Chen, X.; Lin, Y.; Wen, J.-R.; and Han, J. 2023. [Don’t Make Your LLM an Evaluation Benchmark Cheater](https://md.research.elamur.ai/free/en/papers/zhou-2023-don-t-make-llm-evaluation/). arXiv:2311.01964.
39. S.; Makini, S. P.; Lee, S.-H.; Torene, S.; Hatwar, S.; Dehaene, S.; Divic, S.; Ermon, S.; Biderman, S.; Lin, S.; Prasad, S.; Piantadosi, S.; Shieber, S.; Misherghi, S.; Kiritchenko, S.; Mishra, S.; Linzen, T.; Schuster, T.; Li, T.; Yu, T.; Ali, T.; Hashimoto, T.; Wu, T.-L.; Desbordes, T.; Rothschild… — [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://md.research.elamur.ai/free/en/papers/srivastava-2022-imitation-game-quantifying-extrapolating-capabilities/)
40. Student. 1908. The probable error of a mean. Biometrika, 1–25. [find](https://scholar.google.com/scholar?q=Student%2E+1908%2E+The+probable+error+of+a+mean%2E+Biometrika%2C+1%E2%80%9325%2E)
41. Taori, R.; Gulrajani, I.; Zhang, T.; Dubois, Y.; Li, X.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. Stanford Alpaca: An Instruction-following LLaMA model. [https://github.com/tatsu-lab/stanford_alpaca](https://github.com/tatsu-lab/stanford_alpaca). [↗](https://github.com/tatsu-lab/stanford_alpaca)
42. Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023. [LLaMA: Open and Efficient Foundation Language Models](https://md.research.elamur.ai/free/en/papers/touvron-2023-llama-open-efficient-foundation-language/). arXiv:2302.13971.
43. Wang, A.; Pruksachatkun, Y.; Nangia, N.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. R. 2019. [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://md.research.elamur.ai/free/en/papers/wang-2019-superglue-stickier-benchmark-general-purpose/).
44. Red Hook, NY, USA: Curran Associates Inc. Wang, A.; Singh, A.; Michael, J.; Hill, F.; Levy, O.; and Bowman, S. 2018. [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://md.research.elamur.ai/free/en/papers/wang-2018-glue-multi-task-benchmark-analysis/). In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Network…
45. Wang, B.; Deng, X.; and Sun, H. 2022. [Iteratively Prompt Pre-trained Language Models for Chain of Thought](https://md.research.elamur.ai/free/en/papers/wang-2022-iteratively-prompt-pre-trained-language/). In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2714–2730. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
46. Wang, B.; and Komatsuzaki, A. 2021. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. [https://github.com/kingoflolz/mesh-transformer-jax](https://github.com/kingoflolz/mesh-transformer-jax). [↗](https://github.com/kingoflolz/mesh-transformer-jax)
47. Wang, X.; Wei, J.; Schuurmans, D.; Le, Q. V.; Chi, E. H.; Narang, S.; Chowdhery, A.; and Zhou, D. 2023. [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://md.research.elamur.ai/free/en/papers/wang-2022-self-consistency-improves-chain-thought/). In The Eleventh International Conference on Learning Representations.
48. Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Chi, E.; Le, Q.; and Zhou, D. 2022. Chain of thought prompting A Dataset RTE WNLI COPA SST-2 MRPC QNLI CB WiC BoolQ StrategyQA Newsmtsc-mt Newsmtsc-rw NLI4Wills CREPE FOMC NewsMet Hyperparameters [find](https://scholar.google.com/scholar?q=Wei%2C+J%2E%3B+Wang%2C+X%2E%3B+Schuurmans%2C+D%2E%3B+Bosma%2C+M%2E%3B+Chi%2C+E%2E%3B+Le%2C+Q%2E%3B+and+Zhou%2C+D%2E+2022%2E+Chain+of+thought+prompting+A+Dataset+RTE+WNLI+COPA+SST%2D2+MRPC+QNLI+CB+WiC+BoolQ+StrategyQA+Newsmtsc%2Dmt+Newsmtsc%2Drw+NLI)

*Updated 2 November 2026*